User interface control of responsive devices

ABSTRACT

Among other things, first signals are received representing manipulation of a physical feature of a physical device by one or more fingers of a hand of a user. Second signals are received representing tissue electrical activity indicative of and occurring prior to the manipulation. The first signals and the second signals are processed to identify the occurrence of the manipulation. A control signal is sent to a game or other application with which the user is interacting. The control signal corresponds to the identified occurrence of the manipulation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 16/055,777, filed on Aug. 6, 2018, which is a continuation andclaims the benefit of Ser. No. 16/055,123, filed on Aug. 5, 2018, theentire contents of all of which are incorporated here by reference.

This application is a continuation-in-part of U.S. patent applicationSer. No. 16/055,859, filed on Aug. 6, 2018, which is a continuation andclaims the benefit of Ser. No. 16/055,123, filed on Aug. 5, 2018, theentire contents of all of which are incorporated here by reference.

This application is a continuation-in-part of U.S. patent applicationSer. No. 16/055,991, filed on Aug. 6, 2018, which is a continuation andclaims the benefit of Ser. No. 16/055,123, filed on Aug. 5, 2018, theentire contents of all of which are incorporated here by reference.

This application is a continuation-in-part of U.S. patent applicationSer. No. 16/055,123, filed on Aug. 5, 2018, the entire contents isincorporated here by reference.

BACKGROUND

This description relates to user interface control of responsivedevices.

A typical user interface (UI) enables interaction between a human and aresponsive device by interpreting inputs from the human to generateoutputs for the responsive device, by interpreting inputs from theresponsive device to generate outputs for the human, or both. Forexample, one kind of user interface device is a keyboard that receiveshuman keypresses (the inputs from the human) and interprets them togenerate key codes as outputs for responsive devices like computers.Another example of a user interface device is a computer mouse withhaptic feedback that receives signals indicating an alert from acomputer (the responsive device), such as the arrival of an incomingmessage (the inputs from the responsive device), and interprets thesignals to generate vibrations as outputs for the human.

SUMMARY

In general, in an aspect, first signals are received representingmanipulation of a physical feature of a physical device by one or morefingers of a hand of a user. Second signals are received representingtissue electrical activity indicative of and occurring prior to themanipulation. The first signals and the second signals are processed toidentify the occurrence of the manipulation. A control signal is sent toa game or other application with which the user is interacting. Thecontrol signal corresponds to the identified occurrence of themanipulation.

Implementations may include one or a combination of two or more of thefollowing features. The manipulation of the physical feature and thetissue electrical activity indicative of the manipulation occur atdifferent times. The processing of the first signals and the secondsignals includes determining which of the manipulation of the physicalfeature and the tissue electrical activity more accurately representsthe occurrence of the manipulation. The processing of the first signalsand the second signals includes determining that the earlier of themanipulation of the physical feature in the tissue electrical activitymore accurately represents the occurrence of the manipulation. Thesending of the control signal to a game or other application includesposting an event to an event queue of an operating system associatedwith the game or the other application. The processing of the firstsignals and the second signals includes applying the first signals andthe second signals to a classifier.

The training of the classifier is based on first signals representingmanipulations of the physical feature and second signals representingtissue electrical activities. The training the classifier includestraining the classifier repeatedly while the user is interacting withthe game or other application. The training of the classifier includesapplying a dedicated calibration routine before the user begins tointeract with the game or other application. Instructions are receivedfrom the user about the training of the classifier. The processing ofthe first signals and the second signals includes applying the firstsignals and the second signals to a selected classifier among a set ofclassifiers. The selected classifier is selected based on input of theuser. The tissue electrical activity corresponds to a contraction orextension or both of a muscle of the user.

The physical device includes a physical mouse and the physical featureincludes a button or switch of the mouse. The manipulation includes amouse click. The tissue electrical activity occurs at the anterior sideof the wrist of the user. A signal is received indicative of a state ofphysical contact between the user and the physical device, and theprocessing the first signals and the second signals to identify theoccurrence of the manipulation includes taking account of the signalindicative of the state of physical contact. The first signals arereceived as a stream of samples. The second signals are received as astream of samples. The first signals or the second signals or both ourtime stamped. Two or more channels of the second signals are received.Third signals are received from sensors facing the posterior wrist onthe radial side of the user. The second signals are received fromsensors belonging to a tattoo-based or sticker-based component. Thesecond signals are received from a user interface device. The secondsignals are received from implanted sensors.

The second signals are received from sensors on a watch. The physicaldevice and the user interface device are electrically coupled. Thephysical device and the user interface device are mechanically coupled.The classifier is customized for a context in which the game or otherapplication is used. The context includes the model or particular unitof the model of the physical device. The context includes the identityof the user. The context includes the behavior of the user interfacedevice. The context includes style of manipulation by the user. Theeffectiveness of the use of the user interface device by the user ismeasured. Information about the first signals, the second signals, thephysical device, the processing of the first signals and the secondsignals, or the sending of the control signal to the game or otherapplication, or combinations of two or more of those his reported to theuser. A signal is received representing a position or orientation of thehand of the user relative to the physical device. The characteristics ofthe manipulation are determined based on the first signals or the secondsignals or both. The characteristics include the forcefulness of themanipulation. The characteristics include identities of one or morefingers involved in the manipulation. The characteristics include wristrotations. Third signals are received from an IMU and the processingtakes account of the third signals from the IMU.

In general, in an aspect, a user interface device has a sensorconfigured to detect, at a wrist of a human, nerve or other tissueelectrical signals associated with an intended contraction of a muscleto cause a rapid motion of a finger. An output provides informationrepresentative of the nerve or other tissue electrical signalsassociated with the intended contraction of the muscle to an interpreterof the information.

Implementations may include one or a combination of two or more of thefollowing features. The interpreter is configured to interpret theinformation representative of the nerve or other tissue electricalsignals and provide an output indicative of the intended contraction ofthe muscle to cause the rapid motion of the finger. A responsive deviceis coupled to the interpreter and configured to respond to the intendedcontraction of the muscle by an action. The responsive device isconfigured to respond by an action that includes changing an audio orvisual presentation to the human. The rapid motion of the fingerincludes a flick of the finger. The responsive device is configured torespond by an action that includes changing the audio or visualpresentation as if the intended contraction of the muscle correspondedto an invocation of a user interface control. The user interface deviceincludes the interpreter. The rapid motion of the finger has a durationless than one-half second. The rapid motion of the finger includes acontraction and an extension of the finger. The contraction andextension are repeated. The flick of the finger includes a flick up ofan index finger.

In general, in an aspect, a user interface device has a sensorconfigured to detect, at a wrist of a human, nerve or other tissueelectrical signals associated with an intended contraction of a muscleto cause a rotation of a part of the human. An output carriesinformation representative of the nerve or other tissue electricalsignals associated with the intended contraction of the muscle to aninterpreter of the information.

Implementations may include one or a combination of two or more of thefollowing features. The interpreter is configured to interpret theinformation representative of the nerve or other tissue electricalsignals and provide an output indicative of the intended contraction ofthe muscle to cause the rotation. A responsive device is coupled to theinterpreter and configured to respond to the intended contraction of themuscle by an action. The responsive device is configured to respond byan action that includes changing an audio or visual presentation to thehuman. The rotation includes a rotation of a palm relative to an elbow.The responsive device is configured to respond by an action thatincludes changing the audio or visual presentation as if the intendedcontraction of the muscle corresponded to manipulation of a displayedgraphical element. The manipulation of the displayed graphical elementincludes rotating the displayed graphical element. The displayedgraphical element includes a three-dimensional representation of anobject and the manipulation of the displayed graphical element includesrotating the displayed graphical element in three-dimensional space. Thedisplayed graphical element includes a digital camera control and themanipulation of the displayed graphical element includes adjusting thedigital camera control. The displayed graphical element includescharacters of an access code and the manipulation of the displayedgraphical element includes rotating the displayed graphical element toselect characters of the access code. The user interface device includesthe interpreter. The rotation has an angular extent within a range of−180° to 180°.

In general, in an aspect, a user interface device has a sensorconfigured to detect, at a wrist of a human, nerve or other tissueelectrical signals associated with an intended contraction of a muscleto cause a stationary hold of a part of the human. An output carriesinformation representative of the nerve or other tissue electricalsignals associated with the intended contraction of the muscle to aninterpreter of the information.

Implementations may include one or a combination of two or more of thefollowing features. The interpreter is configured to interpret theinformation representative of the nerve or other tissue electricalsignals and provide an output indicative of the intended contraction ofthe muscle to cause the stationary hold. A responsive device is coupledto the interpreter and configured to respond to the intended contractionof the muscle by an action. The responsive device is configured torespond by an action that includes changing an audio or visualpresentation to the human. The stationary hold includes the humanholding a hand in a position. The stationary hold includes a part of thehuman held in a position at the end of another intended contraction ofthe muscle to cause a motion of the part of the human. The responsivedevice is configured to respond by an action that includes continuing achange of the audio or visual presentation until the stationary holdends. The other intended contraction of the muscle is to cause arotation of the part of the human and the stationary hold includes thepart of the human being held in a position during or at the end of therotation. The other intended contraction of the muscle is to cause anindex finger lift and the stationary hold includes the finger being heldin a position during or at the end of the rotation. The responsivedevice is configured to respond to the stationary hold by locking theresponsive device or unlocking the responsive device. The sensor isconfigured to detect another intended contraction of the muscle, theother intended contraction of the muscle to cause a flick of a finger.The interpreter and in which the interpreter is configured to interpretthe information representative of the nerve or other tissue electricalsignals and to provide an output indicative of the combination of theintended contraction and the other intended contraction. The responsivedevice is configured to respond by an action that includes changing theaudio or visual presentation as if the intended contraction of themuscle corresponded to a locked or unlocked state of an applicationbeing executed on the responsive device. The user interface deviceincludes the interpreter. The user interface of claim including acontroller that is at least partly in the user interface device. Theuser interface of claim in which the controller is at least partly in aresponsive device.

In general, in an aspect, a user interface device has a sensorconfigured to detect nerve or other tissue electrical signals associatedwith an intended contraction of a muscle to cause a motion of a part ofa human. An output provides information representative of the nerve orother tissue electrical signals associated with the intended contractionof the muscle to cause an action by a responsive device. A feedbackcomponent is configured to provide feedback to the human indicative ofthe action by the responsive device.

Implementations may include one or a combination of two or more of thefollowing features. The feedback component is part of the user interfacedevice. The feedback component includes a haptic element configured toprovide haptic feedback to the human. The feedback component providesinformation to enable the human to control an intended contraction ofthe muscle in accordance with the feedback. The feedback component isconfigured to receive feedback information from the responsive device.The responsive device includes the feedback component. The responsivedevice includes smart glasses. The feedback includes a visible elementdisplayed by the smart glasses. The feedback includes an elementdisplayed by the responsive device. The feedback includes a soundproduced by the responsive device.

In general, in an aspect, an apparatus includes two or more userinterface devices. Each of the user interface devices has a sensor todetect an input from a human. An output carries informationrepresentative of the input to an interpreter of the information. Afirst one of the user interface devices has a sensor configured todetect, at a wrist of a human, nerve or other tissue electrical signalsassociated with an intended contraction of a muscle. An output carriesinformation representative of the nerve or other tissue electricalsignals associated with the intended contraction of the muscle to aninterpreter of the information. An interpreter is configured to generatean interpreted output based on a combination of the output of the firstone of the user interface devices and an output of another one of theuser interface devices.

Implementations may include one or a combination of two or more of thefollowing features. A responsive device is configured to take an actionin response to the interpreted output. Two or more of the user interfacedevices have sensors configured to detect, at a wrist of the human,nerve or other tissue electrical signals associated with intendedcontraction of two or more muscles. The sensor of one of the userinterface devices includes an inertial measurement unit (IMU). Theinertial measurement unit is calibrated using a calibration technique.The inertial measurement unit is configured to access a valuecorresponding to a reference point. The reference point is independentof the position of the inertial measurement unit. A vector provided bythe inertial measurement unit is compared to a second vector that iscalculated using the value corresponding to the reference point. Thecalibration technique is performed after a pre-defined number of spatialcalculations of the inertial measurement unit. The calibration techniqueis performed in response to a command from the human. There are two ormore responsive devices configured to take respective actions inresponse to the interpreted output of at least one of the user interfacedevices. Two of the user interface devices are wrist-worn user interfacedevices. At least two of the user interface devices have sensors todetect inputs from a single human. At least two of the user interfacesdevices are worn by different humans and are wrist-worn devices.

In general, in an aspect, a user interface device has a sensor to detectnerve or other tissue electrical signals associated with an intendedcontraction of a muscle to cause a motion of a part of a human. Anoutput provides information representative of the nerve or other tissueelectrical signals associated with the intended contraction of themuscle to cause an action by a responsive device. The informationprovided at the output includes raw data representing the nerve or othertissue electrical signals. The output provides the information to aninterpreter external to the user interface device and is configured tointerpret the raw data for use in causing the action by the responsivedevice.

Implementations may include one or a combination of two or more of thefollowing features. The output is coupled to the responsive device. Theoutput is coupled to a communication channel that is also coupled to theresponsive device. The communication channel includes a mesh network.The output is coupled to an intermediary device that is also coupled tothe responsive device. The output is coupled to a socket. The output iscoupled to a distribution system to distribute the output to responsivedevices located at distances from the user interface device greater thancan be reached by Bluetooth communications.

In general, in an aspect, a user interface device has a sensor to detectnerve or other tissue electrical signals associated with an intendedcontraction of a muscle to cause a motion of a part of a human. Anoutput provides information representative of the nerve or other tissueelectrical signals associated with the intended contraction of themuscle to cause an action by a responsive device. The output is coupledto a distribution system to distribute the output to responsive deviceslocated at distances from the user interface device greater than can bereached by Bluetooth communications.

Implementations may include one or a combination of two or more of thefollowing features. The output is coupled to the distribution systemthrough a socket. The output is coupled to a mesh network.

In general, in an aspect, a user interface device has a sensorconfigured to detect, at a wrist of a human, nerve or other tissueelectrical signals associated with an intended contraction of a muscleto cause a motion of a part of a human. An output provides informationrepresentative of the nerve or other tissue electrical signalsassociated with the intended contraction of the muscle to an interpreterof the information.

Implementations may include one or a combination of two or more of thefollowing features. There is a controller. The controller is at leastpartly in the user interface device. The controller is at least partlyin a responsive device. The controller is configured to effect an actionby a responsive device. The controller is configured to effect theaction by altering a presentation by the responsive device. Thecontroller is configured to alter the presentation by unlocking one ormore of the responsive devices. The controller is configured to alterthe presentation with respect to a scrolling function. The controller isconfigured to alter the presentation by presenting a selection. Thecontroller is configured to alter the presentation by presenting a homescreen. The controller is configured to alter the presentation byproviding a visual or audible indication of an output of the controller.The visual or audible indication includes at least one of a change ofcolor, a flashing, a change of size, a change of shape, a change ofappearance, or a combination of any two or more of them. The controlleris configured to generate an output that unlocks a functionality of aresponsive device. The controller is configured to cause the responsivedevice to navigate a menu. The controller is configured to cause theresponsive device to perform a selection. The controller is configuredto cause the responsive device to perform a zoom. The zoom alters apresentation including a display. The controller is configured to causethe responsive device to capture an image, video, or a sound, or acombination of them. The selection is performed on alphanumericalcharacters. The responsive device is configured to operate compatiblywith a peripheral device that can generate an output for controlling theaction of the responsive device. The controller is configured to causethe responsive device to perform an action that can also be caused bythe output generated by the peripheral device. The controller isconfigured not to control the action of the responsive device unless thehuman is in contact with the peripheral device. The peripheral deviceincludes a handheld peripheral device. The action of the responsivedevice is controlled without the human touching the responsive device.The controller is configured to send the output to the responsive devicebased on a Bluetooth round robin. An intermediary routing device managesconnections with the responsive device. The controller is configured tosend the output to the responsive device through the intermediaryrouting device. The responsive device is configured to provide apresentation to a user of the user interface device. The presentationincludes a display or sound or both. A second user interface device isconfigured to detect additional nerve or other tissue electrical signalsof a human and generate data representative of the additional nerve orother tissue electrical signals. One or more interpreters are configuredto make interpretations of the data representative of the additionalnerve or other tissue electrical signals. The controller is configuredto generate the output based on a joint consideration of theinterpretations of the data representative of the nerve or other tissueelectrical signals and the interpretations of the data representative ofthe additional nerve or other tissue electrical signals. The controlleris configured to generate the output based on a separate considerationof the interpretations of the data representative of the nerve or othertissue electrical signals and the interpretations of the datarepresentative of the additional nerve or other tissue electricalsignals. The controller is configured to generate the output based onaudio input. The controller is configured to send the output with a timedelay. The time delay corresponds to a time required for interpretationof the data representative of the nerve or other tissue electricalsignals. The nerve or other tissue electrical signals of the humandetected by the sensor correspond to one or more gestures or intendedgestures. The nerve or other tissue electrical signals of the humandetected by the sensor correspond to one or more motions or intendedmotions of the human. The one or more interpreters are configured toprovide one or more interpretations indicative of the one or moremotions or intended motions of the human. The one or more motions orintended motions of the human include one or more muscle contractions orextensions in an upper extremity of the human.

In general, in an aspect, a user interface device has a sensorconfigured to detect, at a wrist of a human, signals indicative of aunique identity of the human, and an output to provide informationrepresentative of the signals to an interpreter of the information foruse in determining the unique identity of the human.

Implementations may include one or a combination of two or more of thefollowing features. An authentication system generates a verificationoutput with respect to an identity of the human based on the providedinformation. The authentication system is configured to control thehuman's access to a responsive device based on the verification output.The provided information includes biometric signal data and theverification output is based on the biometric signal data. The biometricsignal data includes nerve or other tissue electrical activity sensed atthe surface of the skin of the wrist. The biometric signal data isindicative of wrist acceleration, wrist orientation, or other motion orposition of the wrist of the human. The biometric signal data isprovided during a passive mode of the user interface device. Thebiometric signal data is provided during an active mode of a responsivedevice being controlled by the human through the user interface device.The verification output is generated repeatedly at successive times. Theverification output is generated continuously.

These and other aspects, features, and implementations (a) can beexpressed as methods, apparatus, systems, components, program products,methods of doing business, means or steps for performing a function, andin other ways, and (b) will become apparent from the followingdescriptions, including the claims.

DESCRIPTION

FIGS. 1A, 1B, 3, 9, 10 and 14 are block diagrams.

FIGS. 2, 4, 5, 6A, 6B, 7A, 7B, 8, and 13 are schematics.

FIGS. 11 and 12 are user interface displays.

Here we describe, among other things, features of user interfaces, insome cases features that rely on detected tissue electrical signals (insome cases, nerve or other tissue electrical signals) to be interpretedto generate outputs for the responsive device. User interfaces that relyon nerve or other tissue electrical signals can require less hardware toimplement, can be operated by a wider range of people, and can be moreintuitive than typical user interfaces, among other advantages. Avariety of user interface techniques for controlling responsive devices(and applications running on them) can be implemented based on such userinterfaces, including user interface techniques not possible ordifficult to implement using typical user interfaces.

We use the term “responsive device” broadly to include, for example, anycombination of hardware components or software components or both thatis capable of performing one or more specific actions in response toreceiving a corresponding signal (e.g., an input signal or an outputsignal generated for, sent to, or received by the responsive device).Examples of responsive devices include, but are not limited to,computers, mobile phones, smart glasses, smartwatches, health andfitness monitors, smart speakers, smart lights, smart thermostats, smarthome devices, virtual reality headsets, televisions, audio systems,cameras, repeaters, nodes in a mesh network, application programminginterfaces (APIs) and other devices, and combinations of them. In somecases, a responsive device may comprise multiple responsive devicesacting in parallel or in a chain or both. For example, a node in a meshnetwork (itself a responsive device), upon receiving a signal, mayrepeat or forward the signal (an action) to one or more other nodes(each of which can also be considered a responsive device and at whichan action is taken). In this example, each individual node can beconsidered a responsive device because it performs an action in responseto receiving a corresponding signal. In some cases, two or moreresponsive devices can perform actions in parallel in response tosignals from one or more user interface devices. In some instances, aresponsive device can perform one or more actions in response to two ormore signals from two different user interface devices.

We use the term “action” broadly to include any function or combinationof functions capable of being performed at, by, or on behalf of aresponsive device. Examples of actions include powering the responsivedevice, performing a mouse click, scrolling, exiting a program,controlling presented user interface controls and other elements,adjusting a volume level, transmitting a signal, and saving data, amonga wide variety of others, and combinations of them.

We use the term “user interface” broadly to include, for example, aconceptual technology or capability that can be implemented in anycombination of hardware components or software components and caninterpret one or more inputs from a human to generate one or moreoutputs for a responsive device, can interpret one or more inputs from aresponsive device to generate one or more outputs for a human, or can doboth. A user interface can take on or be implemented in various physicalforms or combinations of physical forms. In some cases, a combination ofcomponents that comprises a user interface can be located physically ina discrete “user interface device” that is distinct from the human andfrom the responsive device.

We use the term “user interface device” broadly to include any one ormore hardware or software components or devices that together at leastpartially implement or provide a user interface by serving as anintermediary between a human and one or more responsive devices. Forexample, a keyboard is a discrete user interface device containingcomponents that interpret human keypresses (the inputs from the human)to generate key codes as outputs for responsive devices like computers.Examples of user interface devices include a keyboard, a mouse, atrackball, a trackpad, smart glasses, wrist worn devices, clickers,augmented reality and virtual reality controllers, and head-trackingdevices, among a wide variety of others, and combinations of them.

In some cases, a device may simultaneously be considered both a userinterface device and a responsive device. For example, a smart watch canbe considered a user interface device because it can serve as anintermediary between a human and a responsive device such as a smarttelevision. The smart watch can also be considered a responsive devicebecause upon receiving an input signal representative of a human touch,the smart watch can itself perform actions such as opening an email orsending control signals to other smart devices.

In some cases, the combination of components that comprises the userinterface can be physically located entirely within the responsivedevice, without the need of a distinct user interface device to serve asan intermediary. For example, in some cases, a television can beoperated without the need of a remote (user interface device) to serveas an intermediary between the user (human) and the television(responsive device). In this example, the human can operate thetelevision (responsive device) directly by pressing buttons on thetelevision because the user interface is entirely located within theresponsive device. Similarly, a combination of components that comprisea responsive device can be physically located entirely within a userinterface device.

In some cases, the combination of components that comprises the userinterface can be physically located within two or more discrete userinterface devices, two or more responsive devices, or a combination ofone or more discrete user interface devices and one or more responsivedevices.

In some cases, a user interface may include one or more interpretersthat perform one or more interpretation steps in order to interpret oneor more inputs from one or more humans to generate one or more outputsfor one or more responsive devices. In some implementations, each of theinterpreters may interpret inputs from a responsive device to generatean output for a human. Or each of the interpreters may perform bothinterpretations of inputs from one or more humans and interpretations ofinputs from a responsive device. For example, a user interface mayinclude a first interpreter to interpret biological inputs (such asnerve or other tissue electrical signals) from a human to generateelectrical signals or data representative of the biological inputs fromthe human. In some cases, biological inputs from the human can includebiological signals indicative of vital signs or physical exercise suchas heart rate, respiratory rate, skin capacitance, oxygen saturation ofblood, a number of steps taken, or a number of calories burned, amongother things. The user interface may include a second interpreter (i.e.,in series with the first interpreter) to interpret the generatedelectrical signals or data representative of the biological inputs fromthe human to generate a corresponding output (such as a control output)for the responsive device. In some instances, elements of an interpretercan be distributed among one or more user interface devices or one ormore responsive devices, or combinations of them.

We use the term “interpret” broadly to include, for example, anydetermination of one or more outputs that depends on, corresponds to, isbased on, translates, maps, transforms, or is determined by one or morereceived inputs. In some cases, interpretation is performed at alow-level, with the determined outputs being substantially similar tothe received inputs. For example, a sensing circuit that outputsdigitized or compressed electrical signals or data corresponding toreceived biological inputs is said to “interpret” the biological inputs.In some cases, interpretation comprises determining a binary outputdependent on whether or not an aggregate amount of received biologicalinputs is greater than a threshold amount, without requiring theclassification of the inputs as a specific gesture. In some cases,interpretation is performed at a higher level such as a classifier(interpreter) used to classify (interpret) the inputs as aninterpretation selected from among a set of possible interpretationsbefore generating an output corresponding to the selectedinterpretation. For example, an interpreter that receives as inputs IMUsignals from a wrist-worn user interface device may classify the signalsas an “N-Degree Palm Rotation” interpretation and generate acorresponding output signal for a computer (a responsive device) to usein controlling an element presented on a display of the computer. Insome cases, multiple interpretations of corresponding inputs may beaggregated by a controller to generate an output signal corresponding tothe aggregated interpretations. In both of these examples, theinterpreter is said to “interpret” the IMU signals. Interpretations canbe characterized by a wide variety of factors. For example,interpretations can include actions, motions, sounds, gestures,intentions, eye movements, actions associated with responsive devices(such as a mouse click or a keyboard keypress, to name just two),physical activities, static holds, pauses, digitized raw signals,compressed raw signals and others.

As shown in FIG. 1A, an example user interface 100 can be implemented inhardware components 103A, 103 B and software components 104A, 104 B. Thesoftware components include machine readable instructions stored inmemory or storage (which can be part of the hardware 103A, 103B) andexecutable by one or more processors (which also can be part of thehardware 103A, 103B). When executed by a processor the instructions caneffect one or more interpretations of one or more inputs 114 from one ormore humans 110 to generate one or more outputs 116 for one or moreresponsive devices 112. In the configuration depicted in FIG. 1A, theuser interface 100 is physically implemented entirely within a discreteself-contained user interface device 102 that is distinct from the human110 and the responsive device 112.

A wide variety of inputs 114 can be provided by the human 110 includingelectrical signals, motions of parts of the body, thermal inputs,perspiration, sounds, breath, and other physiological actions (voluntaryand involuntary). In some examples, the inputs can include nerve andother tissue electrical signals and other biological signals that can bedetected by one or more sensors (e.g. sensor 124) that are placed on ornear the human. In some examples, the inputs can includeelectromyography (EMG) signals collected at, or directly above, the siteof a muscle, morphed EMG signals that have propagated through the bodyfrom a source to a propagation site distinct from the site of the muscleor the skin directly above it, motion artifacts from tendon movement, orany combination of them (e.g. in a bulk signal). In some cases, theinputs can include biological signals indicative of vital signs orphysical exercise such as heart rate, respiratory rate, skincapacitance, oxygen saturation of blood, a number of steps taken, or anumber of calories burned, among other things.

We use the term “sensor” broadly to include, for example, any devicethat can sense, detect, or measure a physical parameter, phenomenon, oroccurrence. Examples of sensors include IMUs, gyroscopes, moisturesensors, temperature sensors, visible light sensors, infrared sensors,cameras, audio sensors, proximity sensors, ultrasound sensors, haptics,skin impedance pressure sensors, electromagnetic interference sensors,touch capacitance sensors, external devices and other detectors, andcombinations of them.

In our discussion, we often use, as an example, user interfaces thatreceive, at least in part, nerve or other tissue electrical signals thatare detected at the wrist by electrical sensors. In some cases, theelectrical sensors are located at a posterior aspect of the wrist (e.g.the top of the wrist), allowing for incorporation of the sensors intowatches, fitness trackers, and other devices having rigid or flexiblebodies. In some cases, having the electrical sensors at only theposterior aspect of the wrist can obviate the need for distributing thesensors circumferentially around the wrist, such as in a wrist strap orbracelet. Technologies for detecting and using nerve or other tissueelectrical signals or other tissue electrical signals at any part of thebody, especially the wrist, are described in U.S. application Ser. No.15/826,131, filed on Nov. 29, 2017, the entire contents of which areincorporated here by reference.

In the example of FIG. 1A, the user interface device 102 includes twointerpreters 106A, 106B. The first interpreter 106A is included in thesensor 124 and is implemented using a combination of hardware components103A and software components 104A. The first interpreter 106A receivesbiological inputs 114 (such as nerve or other tissue electrical signals)from the human 110 and interprets them to generate electrical signals ordata 126 representative of the biological inputs 114. In this example,the second interpreter 106B is included in a controller 128 and isimplemented using a combination of hardware components 103B and softwarecomponents 104B. The second interpreter 106B receives the electricalsignals or data 126 representative of the biological inputs 114 from thehuman 110 and interprets them. In order to interpret the electricalsignals or data 126, the second interpreter 106B stores informationabout possible interpretations 108A-D. Upon receiving the electricalsignals or data 126, the second interpreter 106B classifies theelectrical signals or data 126 as corresponding to one of theinterpretations among the possible interpretations 108A-D. Thecontroller 128 aggregates the interpretations made by the secondinterpreter 106B and generates an output 116 for the responsive device112 depending on a variety of contextual factors including, but notlimited to, the timing of the interpretations, the type of responsivedevice 112, and an operational status of the responsive device, orcombinations of them, among other things.

In examples discussed here, interpretations may include identifiedgestures or identified intentions for gestures. For example, based onelectrical signals 126 generated by an IMU sensor 124, the secondinterpreter 106B may classify the signals 126 as corresponding to an“N-Degree Palm Rotation” gesture 108A. Similarly, based on electricalsignals 126 generated by an electroneurography (e.g., nerve or othertissue electrical signal) sensor 124, the second interpreter 106B mayclassify the signals 126 as an “Intended Index Finger Flick Up” gesture108B. Another interpretation could be a “fist formation” 108C.Interpretations may also be lower-level. For example, based onelectrical signals 126 generated by the first interpreter, the secondinterpreter 106B may simply interpret the signals as “Digitized RawSignal” 108D and generate an output 116 that is similar in its level ofabstraction to the raw electrical signals 126.

Classifying the electrical signals 126 as corresponding to aninterpretation selected from the interpretations 108A-D can be doneusing a variety of classification techniques. For example, in somecases, classification (interpretation) of nerve or other tissueelectrical signals (our data corresponding to the signal) as an“Intended Index Finger Flick Up” gesture can be implemented using asingle threshold value. The average voltage amplitude of gestures can beused to train and select generic threshold values manually orautomatically through machine learning or other algorithms. In somecases, classification can involve signal processing of the electricalsignals 126 and the implementation of techniques such as logisticregression, decision trees, support vector machines, or neural networks.In some cases, accurate classification of the electrical signals 126 canbe performed without requiring a user to complete a dedicatedcalibration process prior to operating the user interface device 102.For example, the second interpreter 106B may include a generalclassifier that is satisfactorily accurate for most people without theneed for personalized calibration prior to operation of the userinterface device 102. In some cases, personalization of the generalclassifier (e.g. by tuning parameters of the general classifier) can beperformed in real time as the user operates the user interface device102.

A wide variety of other interpretation techniques can be applied inaddition to or other than classification. For example, machine learningtechniques using features such as entropy, root-mean square, averagevalue, frequency-based analysis, pattern matching, log detect, spatialanalysis from multiple electrode channels, sensor fusion with IMU andother inputs, time-variance of user behavior, slope of amplitudechanges, and duration and shape of waveforms can be applied, among otherthings.

After classifying the electrical signals or data 126 as aninterpretation selected from the interpretations 108A-C, the userinterface device 102 can then send to a responsive device 112 an output116 corresponding to the classified interpretation. Outputs 116 for theresponsive device 112 can include data or commands for controllingactions 122 of the responsive device 112. Outputs 116 can also includeraw digitized information for a responsive device 112 to utilize as itwishes.

More generally, outputs for the responsive device can comprise a widevariety of forms and modes. For example, outputs can be in the form ofelectrical signals, data, commands, instructions, packets,electromagnetic waves, sounds, light, and combinations of them, amongothers. Outputs can conform to a wide variety of standard or proprietarycommunication protocols.

The responsive device 112 may include an interpreter 130 forinterpreting the output 116 in order to perform a corresponding action122. Upon receiving the output 116 for the responsive device 112, theresponsive device 112 interprets the output 116 and performs thecorresponding action 122. The action can comprise a wide variety offorms and modes. An action can include a physical motion, a sound, adisplay, a light, or a presentation or manipulation of textual,graphical, audiovisual or other user interface elements on a displaydevice, or combinations of them, among other things.

The user interface 100 of FIG. 1A can be configured to operate as anopen control loop or a closed control loop. In an open control loop, theaction 122 performed by the responsive device 112 does not providefeedback to the user that directly influences the human's inputs.However, as discussed previously, a user interface may be capable notonly of receiving inputs from a human to generate outputs for aresponsive device, but may also receive inputs from the responsivedevice to generate outputs for the human. In some cases, it is possiblefor a user interface to include closed loop control, using feedbackcontrol elements 132.

For example, consider a mouse a with a haptic feedback element. Themouse is a user interface device 102 that receives motion and clickinginputs 114 from a human 110 and interprets the inputs to generate acommand (output 116) for a computer (responsive device 112) to move acursor or perform a mouse click (action 122). Depending on contextualfactors, in response to receiving a mouse click command (output 116),the computer (responsive device 112) may send a vibration command(action 122) to the mouse (the user interface device 102). The vibrationcommand can be interpreted and acted upon by the mouse's haptic feedbackelement (feedback control elements 132), thus providing vibrations(outputs for the human 134) to guide the user's further operation of theresponsive device 112 or other responsive devices through the userinterface device 102 or other user interface devices. While this exampleshows the feedback control elements 132 in the user interface device102, in some cases, the feedback control elements 132 may be included inthe responsive device 112, or in a combination of the user interfacedevice 102 and the responsive device 112.

As mentioned previously, a user interface can be thought of as aconceptual technology that can take on various physical forms. FIG. 1Bshows a configuration of a user interface 100, in which the userinterface 100 is located (as indicated by the dashed lines) partially ina responsive device 112 and partially in a discrete user interfacedevice 102. Similarly to the example shown in FIG. 1A, a human 110generates biological inputs 114 that are detected by the sensor 124 andinterpreted by the first interpreter 106A to generate output signals 126representative of the biological inputs 114. However, in thisconfiguration the user interface device 102 does not include thecontroller 128 with the second interpreter 106B. Rather, the controller128 with the second interpreter 106B is located in the responsive device112. The output signals 126 can be transmitted to the controller 128 inthe responsive device 112 by sending electrical signals over a wiredconnection or by sending wirelessly transmitted radio frequency signals.The output signals 126 are received by the controller 128 within theresponsive device 112, and the second interpreter 106B interprets theoutput signals 126 to generate an output 116 for the responsive device112. Upon receiving the output 116 for the responsive device 112, theresponsive device 112 interprets the output 116 using the interpreter130 and performs the corresponding action 122. In some cases, in aclosed loop user interface configuration, the action 122 may includesending a signal to feedback control elements 132 in order to generateoutputs for the human 134.

FIGS. 1A and 1B show only example configurations and architectures ofuser interfaces and components and devices used to implement them.Configurations may include user interfaces having additional sensors,additional interpretation steps, multiple user interface devices,multiple responsive devices, and other combinations of components anddevices.

As shown in FIG. 2, a user interface 100 (for example, the userinterface of FIG. 1A or FIG. 1B) can be used to operate, control,command, or otherwise operate any kind of responsive device 112 such asa computer or a mobile phone, to name two. In some cases, the responsivedevice 112 can perform actions that affect an on-screen display200A-200D of the responsive device 112 at different moments. At a firstmoment, the on-screen display of the responsive device 112 may show alock screen 200A. To unlock the screen, a user (human 110) wearing awrist-worn user interface device 102 may rotate his palm 180 degrees (orsome other angular amount within a range of 20 degrees to 360 degrees,for example) back and forth three times (or some other number of timessuch as a number of times between one and five) within a given timeframe. The time frame may be between 0 and 3 seconds to correspond tothe average time it takes a human to rotate his palm 180 degrees backand forth three times as quickly as possible. For example, the timeframe may be between 0 and 1 seconds, 1 and 2 seconds, or 2 and 3seconds, or other time frames. We use the term “rotate the palm” torefer, for example, to a motion in which a plane in which the palm liesis rotated about an axis that lies along the length of forearm. In somecases, rotation of the palm entails manipulation of the two bones of theforearm relative to the elbow.

Referring to the user interface configurations depicted in FIGS. 1A and1B, the motion of the palm is a biological input 114 that is interpretedby an IMU sensor 124 included within the user interface device 102 togenerate electrical signals 126 representative of the back and forthrotational motion. In the example being discussed, the controller 128then classifies the electrical signals 126 as six consecutive “N-DegreePalm Rotation” gestures 108A, where N=180, −180, 180, −180, 180, and−180 (corresponding to three back and forth palm rotations). If thecontroller 128 determines that the three back and forth palm rotationsoccurred within the given time frame (e.g. 0-1 seconds, 1-2 seconds, or2-3 seconds), the controller 128 generates an output 116 for theresponsive device 112 commanding it to unlock (action 122) the lockscreen 200A. In this example, the controller 128 combines (aggregates)six interpretations (i.e. “N-Degree Palm Rotation” gestures) of theinterpreter 106B to generate a single output 116 for the responsivedevice 112. In general, the controller 128 can combine any number ofsimilar or different interpretations 108A-C of the interpreter 106B intoan action 122 depending on various contextual factors including, but notlimited to, the timing of the interpretations, the type of responsivedevice 112, the status of the on-screen display 200A-D, the startingposition or orientation of the hand, and combinations of them, amongother things.

Upon unlocking (action 122) the lock screen 200A, the responsive device112 shows a home menu 200B on the on-screen display. The home menu 200Bincludes selectable icons 202A-D representing applications that can berun by the responsive device 112. The home menu 200B includes a clock204 and status indicators representing information about the responsivedevice 112 such as a Bluetooth connection 206, a network strength 208,and a battery life 210. In FIG. 2, the “3D Model” icon 202A currentlyhas the focus for selection, as indicated by a thicker circular border.If a user (human 110) wishes to toggle through the icons, one by one,the user can rotate his palm clockwise once each time he wishes totoggle to the next icon. The motion (input 114) is interpreted togenerate electrical signals or data 126, and the electrical signals ordata 126 are classified as an “N-Degree Palm Rotation” gesture 108A.However, because the “N-Degree Palm Rotation” gesture 108A is identifiedwhile the user is working on the home screen 200B instead of on the lockscreen 200A (in other words, the context is different), the output 116generated for the responsive device 112 is different. Unlike while theuser is working on the lock screen 200A, if the interpreter 106Binterprets the electrical signals 126 as an “N-Degree Palm Rotation”,the controller 128 does not wait to identify if three back and forthpalm rotations are performed. In this case, if the interpreter 106Binterprets the electrical signals 126 as an “N-Degree Palm Rotation”, ifN is greater than a pre-specified threshold amount in the clockwisedirection (e.g., 10-15 degrees, or another pre-specified thresholdamount in the range of 2 degrees to 30 degrees), the controller 128immediately outputs a command to the responsive device 112 to scroll tothe next icon in the order of “3D Model”, “Manual”, “Scan”, and“Settings” (action 122). Similarly, the user (human 110) can rotate hispalm counterclockwise (in accordance with the same or a differentthreshold amount) to toggle the icons 202A-D in a counterclockwisemanner.

In some cases, if many icons are present on the on-screen display 200B,it can be uncomfortable for the user to rotate her palm 10-15 degreesevery time she wants to move to the next icon, and if the rotationthreshold is lowered too much, for example to 1-2 degrees, it could bechallenging for a user to control. In some implementations, the userinterface 100 can be configured so that the user can rotate her palmbeyond the specified amount in the clockwise or counterclockwisedirection (e.g. 10-15 degrees) once and hold her hand in that positionto continue auto-scrolling in that direction. Likewise, to stopauto-scrolling, the user can rotate her palm once beyond the specifiedamount in the opposite direction.

A wide variety of other gestures can be expressed by a human user in theform of palm rotations. A variety of combinations of the number ofrotations, the angular amounts of the rotations, and the durations ofholding of the hand in a position can be used to represent a range ofintentions and gestures. Various numbers of successive rotations in thesame direction or in opposite directions and in any sequence could beused to express a variety of different intentions or gestures. A varietyof different amounts of rotation could be used to express differentintentions or gestures. And various combinations of amounts ofrotations, numbers of successive rotations, and directions of rotationscould be used to express different intentions or gestures. Rotations ofthe palm could be combined with other motions of one or more of thefingers or hand or both to express corresponding intentions or gestures.The hand motions could include pitch motions or yaw motions orcombinations of them around the wrist relative to the forearm and suchhand motions could be of various angular amounts, repetitions,directions, and combinations of them, in conjunction with palmrotations. Combinations of motions to express gestures or intentionscould therefore range from simple to complex and could correspond to awide variety of actions by the responsive devices. The angularorientation of the wrist and forearm can be used to differentiatebetween gestures. For example, pointing the hand and arm up isdetectable by an IMU and can be mapped to an interface hierarchy commandsuch as a home or menu command. Pointing the hand and arm diagonalupwards or downwards can similarly be used to navigate interfaces.Pointing down can be used to trigger a calibration state for the IMU forpurposes of hand spatial tracking or resetting a user experience. Insome cases, palm rotations can be combined with other gestures, such asflicking an index finger up, to correspond to a single action by theresponsive device.

Referring again to FIG. 2, once an icon, say the “3D Model” icon 202A,has been toggled, to cause the responsive device to run thecorresponding application (for example, the 3D modeling application)(action 122), the user (human 110) may flick his index finger up. Anindex finger flick up may comprise any rapid extension of the indexfinger, for example, beyond its anatomical resting position or beyondsome other previously held position. For example, an index finger flickup may comprise the tip of the index finger moving within a range of 0-5cm within a time frame of 0-500 ms. Other ranges of movement and timeframes also could be considered finger flicks up.

Extensions and contractions of the index finger, as well as movements ofany other body part, can themselves comprise extensions and contractionsof various muscles. In some cases, remaining motionless also comprisesthe contraction of various muscles (i.e. in an isometric manner). We usethe term “contraction” in the context of muscles broadly, to include forexample, any activation of tension-generating sites in muscle fibers. Weuse the term “extension” in the context of muscles broadly, to includefor example, any relaxation of tension-generating sites in muscle fibersor lengthening of the muscles.

In this example, the user interface device 102 on the user's wristdetects nerve or other tissue electrical signals (inputs 114), anelectroneurography (e.g., nerve or other tissue electrical signal)sensor 124 generates corresponding electrical signals 126, aninterpreter 106B classifies the electrical signals 126 as an “IntendedIndex Finger Flick Up” gesture 108B, the controller 128 sends acorresponding output command 116 for the responsive device 112 to openthe 3D modelling application (action 122), and the responsive device 112acts accordingly, for example, by launching the application whose iconhas the focus.

Although in this example, we are referring to a rapid motion of a singlefinger as a gesture to trigger an action, rapid motions of otherindividual fingers (e.g., another finger or thumb) or combinations oftwo or more fingers could also be used as corresponding gestures. Thespeeds of motion and the extent of motion and combinations of them couldbe used to define different gestures. Combinations of motions ofparticular fingers or combinations of fingers with speeds of motion andextensive motion could provide additional defined gestures.

A user does not need to physically perform a gesture (for example causean actual flicking motion of her index finger) in order to generateinput that can be correctly classified as if it were a physical gestureby the user interface 100. Because nerve or other tissue electricalsignals can be detected, interpreted, and acted on, irrespective ofmuscle contraction or extension, the intention of a user (for example,to effect a gesture such as the flick of an index finger) can becaptured by the nerve or other tissue electrical signals, even in caseswhere the user is unable to contract or extend the muscles necessary toperform a gesture due to conditions such as Amyotrophic LateralSclerosis (ALS), stroke, or amputation.

Upon invoking the “3D Model” icon 202A in response to a correctinterpretation of a user's index finger flick up as indicating a gestureto launch the application, the responsive device 112 shows a 3Dworkspace screen 200C on the on-screen display. In some cases, on thisscreen, rotations of the palm (inputs 114) can be interpreted togenerate commands (outputs 116) to the responsive device 112 to rotate(or otherwise reorient) a presented assembly 212 in 3D space (action122). In other words, gestures (palm rotation) similar to the onesdescribed with respect to screen 200B, can be interpreted (based on thecontext of the rotation occurring with respect to a different screendisplay) as intending a different action by the responsive device, inthis case rotation of a presented assembly. In addition, while theapplication is presenting the display 200C, nerve or other tissueelectrical signals corresponding to an index finger flick up (inputs114) can be interpreted (contextually) to generate a command (output116) to the responsive device 112 to present an exploded view of thesame assembly 214 as shown on screen 200D.

At any time, a user (human 110) can return to the home screen 200B byrotating her palm 180 degrees, for example, so that it is facing upwardand by holding her palm in that position for a certain amount of time,for example, within a range of 1-3 seconds (or other ranges). By servingas a shortcut to the home screen 200B, this gesture allows for quicklyswitching between applications being run on the responsive device 112and for easily navigating the menu hierarchy of the responsive device112.

As shown in FIG. 2, a user interface can be used to control a responsivedevice based on a sequence of gestures or a combination of gestures thatcan be as simple or as complex as necessary to enable a full, rich,robust, or deep control of one or more (including every) feature,function, or capability of any kind of recognition device. In somerespects, the library of gestures, sequences of gestures, andcombinations of gestures can not only be sufficient to control everyaspect of a recognition device, but in some circumstances the individualgestures, sequences of gestures, and combinations of gestures can besimpler, easier, quicker, and more robust in their control of therecognition device than can be provided by typical user interfaces.

The gestures can be mapped as shortcuts, used to trigger macros, ormapped to hotkeys of a responsive device in a universally accessible ora context-specific way. For example, an index finger flick up can beused to take a picture when the responsive device is in camera mode. Thesame gesture can be used to emulate a computer click when the responsivedevice is in a desktop navigation mode. Lifting a finger and holding for1-5 seconds can be used as a universal gesture to lock the device fromfurther commands except the unlock command which could be executed withthe same gesture.

The example user interface for operating a responsive device, asdescribed in relation to FIG. 2, could apply to the launching andrunning by the responsive device of a wide variety of applications inaddition to, or instead of, the “3D Model”, “Manual”, “Scan” and“Settings” applications.

As mentioned above, additional gestures, both intentional andunintentional, may be performed to operate a responsive device 112, andthe controller 128 can be configured to generate different outputs 116in response to the same interpretations. For example, while “N-DegreePalm Rotation” and “Intended Index Finger Flick Up” gestures have beendescribed, other finger, hand, or other body gestures may beinterpreted. In particular, for user interfaces, it can be useful to usegestures that are performed infrequently by a user when he is notoperating the responsive device 112 in order to avoid generatingunintended actions of the responsive device 112. Supplemental oralternative gestures may include flicking an index finger down, forminga fist, remaining motionless (e.g. a pause or a hold), and abducting oradducting the fingers, or combinations of those.

As illustrated in FIG. 13, in one supplemental gesture, the user startswith the hand in a resting position 1300, subsequently lifts his indexfinger up 1302, and rotates his palm 1304A within a time period of 0-3seconds. While a clockwise rotation is shown, the rotation of the palmmay also be in a counterclockwise direction.

Also illustrated in FIG. 13, in one supplemental gesture, the userstarts with the hand in a resting position 1300, subsequently lifts hisindex finger up 1302, and raises his hand to point diagonally up 1304Bwithin a time period of 0-3 seconds. While the user is shown to raisehis hand to point diagonally up, in some cases, the supplemental gesturemay comprise the user lowering his hand to point diagonally down.Raising or lowering the hand to point diagonally up or down can beperformed, for example, by bending an arm at the elbow.

In some cases, a user interface device 102 can receive input signals(e.g. nerve or other tissue electrical signals or IMU signals)corresponding to one or both of these gestures and control acorresponding action of a responsive device 112 in accordance with theuser interfaces 100 of FIGS. 1A and 1B. In some cases, the correspondingaction of the responsive device 112 may depend on contextual factorssuch as the starting position or orientation of the hand and arm. Forexample, if one of the supplemental gestures described is performed withthe hand and arm starting in a substantially horizontal orientation (asshown in resting position 1300), one corresponding action may beperformed. However, if the same supplemental gesture is performed withthe hand and arm starting in a substantially vertical orientation, adifferent corresponding action may be performed.

In addition to the benefits of providing inputs to a user interfacealready described in relation to FIG. 2, interpreting nerve or othertissue electrical signals occurring at, for example, the wrist (that is,“upstream”) as corresponding to muscle contraction, extension, or boththat the nerve or other tissue electrical signals will cause has otherbenefits. One of the benefit is in reducing the amount of time requiredfor the user interface to determine the occurrence of a musclecontraction, extension, or both compared to user interfaces that requirea user interface device, such as a mouse, a game controller, or anotherperipheral device, to be physically actuated by muscle contractions,muscle extensions, or both of the user. By detecting nerve or othertissue electrical signals, it is possible to interpret or register auser's intention to perform an action (e.g., make a gesture) 20-150 msbefore a physical click on a mouse or game controller would beregistered. This is especially important in applications such ase-sports or games or other applications, where users may perform up to300 computer actions per minute (APM).

A common performance measurement for e-sports or performance sports ingeneral is reaction time. Reaction time corresponds to the amount oftime the brain takes to process a stimulus and execute a command to apart of the body to physically react such as by clicking a mouse.Typical reaction time can be 100 ms-400 ms. The amount of time saved inthe interpretation of the user's intention can be significant withrespect to the amount of time to perform the intended action of theresponsive device to be controlled. In some examples, a user interfacedevice that senses nerve or other tissue electrical signals can beplaced on the anterior side of the wrist instead of the posterior sideof the wrist to achieve improved detection of an intention to perform anindex finger flick or a mouse click, for example. In some examples, theuser interface device may be worn on the forearm for increased comfort.In some examples, to reduce false positives, a touch sensor may beincluded on a mouse or game controller so that the user interface devicesends an output to the responsive device to perform a mouse click onlywhen a user's hand is placed on the mouse or game controller.

In addition to the benefits of providing inputs to a user interfacealready described in relation to FIG. 2, interpreting nerve or othertissue electrical signals occurring at, for example, the wrist (that is,“upstream” or proximal) as corresponding to muscle flexion or extension,that the nerve or other tissue electrical signals will cause, may haveother benefits. One of the benefits is a reduction in time required forthe user interface to interpret (e.g., predict) the occurrence of amuscle flexion or extension, compared to interfaces that require a userinterface device, such as a mouse, a game controller, or anotherperipheral device (e.g., a physical peripheral device), to be physicallyactuated by muscle flexion or extension (or any muscle contractions) ofthe user. By detecting nerve or other tissue electrical signals“upstream”, it is possible to interpret or register a user's intentionto perform an action (e.g., make a gesture) 20-150 ms before acorresponding physical click on a mouse or game controller would beregistered. On average, the prediction of the physical click couldprecede the actual click by, for example, about 30 milliseconds. This isespecially important in applications such as e-sports or games or otherapplications, where users may perform up to 600 computer actions perminute (APM).

A common performance measurement for e-sports or performance sports ingeneral is reaction time. Reaction time corresponds to the periodrequired for brain processing of a stimulus and the execution of adecision (command) to move a part of the body (e.g., limb or finger) inorder to complete a physical action, such as clicking a mouse device.Typical reaction time can be 100 ms-400 ms. The reduction in overallreaction time, through a predictive interpretation method, cansignificantly impact the intended functioning of the responsive deviceto be controlled, in a time sensitive task.

In the field of competitive video games, especially, the outcome of amatch can depend upon millisecond differences in reaction time. Forexample, in a first-person shooter (FPS) game, the winner of aone-on-one engagement is the player who is able to shoot first and hitthe enemy target. In some FPS games, a user whose input is detected amere 10 milliseconds after a competitor's input might result in a loss.In real time strategy games (RTS) like StarCraft 2, the victor of amatch tends to be the player with the most actions per minute (APM). Aplayer with superior reaction time, with lower-latency input, is likelyto have a higher APM, and therefore win.

In some examples, a user interface device that senses nerve or othertissue electrical signals can be placed on the anterior side of thewrist instead of the posterior side of the wrist to achieve improveddetection of an intention to perform an index finger flick or a mouseclick, for example. In some examples, the user interface device may beworn on the forearm for increased comfort. In some examples, to reducefalse positives, a touch sensor may be included on a mouse or gamecontroller so that the user interface device sends an output to theresponsive device to perform a mouse click only when a user's hand isplaced on the mouse or game controller.

As shown in FIG. 14, in some implementations of the user interfacesystem 1400, a player of a game (e.g., a game console, electronic game,or game software running on a general-purpose device) or generally auser 1401 of a responsive device 1402 typically uses one or more fingersor a hand to manipulate physical features of a mouse (or trackball,touchpad, game controller, or other similar device) or another physicaldevice 1404 to indicate an action to the game or other application. Wesometimes use the word “game” broadly to refer to a game or any otherapplication. We sometimes use the word “mouse” to refer broadly to, forexample, a traditional mouse or to any other physical user interfacedevice.

In addition to the physical device 1404, a user interface device 1403can be worn at the same time on the user's wrist or another part of theuser's arm associated with the hand or fingers that are used tomanipulate the physical device. The user interface device can includetissue electrical (e.g., biopotential) signal sensors 1405 as describedearlier. In some cases, the signal sensors can face the anterior side ofthe wrist.

Signals 1406 from the user interface device and signals 1410 from thephysical device can be transmitted to an interpreter 1408 wirelessly(for example on a Bluetooth low energy channel) or by wire or acombination of the two.

In some implementations, the signals 1410 take the form of a stream ofbinary signals (e.g., on and off) representing, for example, the clicksof a mouse. In some cases, the signals 1410 can take the form of astream of non-binary signals (e.g., representing signal levels within arange). The signals 1410 therefore represent, for example, the states ofphysical switches or physical variable controls belonging to thephysical device. The signals 1410 can be sampled frequently (say, forexample, at a rate of 10 Hz). And the physical device or the interpretercan timestamp each sample and store it for later analysis and use.

Similarly, in some cases, the signals 1406 take the form of a stream ofbinary signals or analog signals sampled at a similar rate. The samplesrepresent one or more tissue electrical signals detected at the skin ofthe user. The stream of signals 1406 can include raw signals, processedsignals, or interpretations of the raw signals for use by theinterpreter. The physical device or the interpreter can timestamp eachsample and store it for later analysis and use. Although FIG. 14 showsthe interpreter as a separate device, as discussed earlier, theinterpreter functions can be performed either in a separate device orthe physical device in the user interface device, in the responsivedevice, or in combinations of any of them.

One function of the interpreter 1408 is to provide signals 1412 to theresponsive device 1402 in a format and in substance corresponding tocommands or information expected by the responsive device concerning theinteraction between the user and the responsive device. For example, ifthe responsive device is a general-purpose computer running a game, anexpected signal from the interpreter could be one indicating that theuser has clicked the mouse to indicate an action that should be taken onthe game.

Traditionally, in a user interface system that includes a physicaldevice, and does not include a user interface device 1403, theinterpreter provides the signals 1412 directly to the responsive devicebased on the signals 1410 received from the physical device. Forexample, when a user clicks a mouse, the interpreter sends a mouse eventto the operating system of the responsive device.

In the user interface system 1400 shown in FIG. 14, the interpreter mayor may not simply pass the mouse click events through from the physicaldevice to the responsive device. By applying a classifier to the tissueelectrical signals, the interpreter may be able to determine that amuscle contraction is imminent, to predict that a mouse click is aboutto occur, and to make that prediction some small amount of time, forexample, 30 ms more or less ahead of the time when the signal from thephysical device indicates that the mouse click has actually occurred. Insuch cases, the interpreter can send a mouse click event to theoperating system of the responsive device earlier than it wouldotherwise have done so. As a result, the user's interaction with anexperience in playing the game or other application is enhancedsignificantly.

In some implementations, the interpreter can use the earlier to occur ofthe tissue electrical signals and the mouse signals to infer theoccurrence of a mouse click and provide the mouse click event to theresponsive device based on the earlier to occur. In effect, theinterpretation of the tissue electrical signals can enable accuratepredictions of the mouse clicks earlier than the actual mouse clicks,making the game player a better player and enhancing the gameexperience.

All tissue electrical signals, mouse clicks, and their correspondingtimestamps can be stored in the physical device, the user interfacedevice, the interpreter, or the responsive device, or combinations ofthem, for later analysis and use.

In some implementations, the determination that a mouse click is aboutto occur can be made by a classifier 1414 (or other machine learning orartificial intelligence process) running as part of the interpreter, thephysical device, the user interface device, the responsive device, or acombination of them. When the classifier is operating in the userinterface device, for example, the classifier, parameters for itsoperation, and its configuration can be downloaded wirelessly or througha wired USB connection, for example, from the interpreter and 10 beupdated as improvements are made to the classifier or additionaltraining data becomes available. Other settings for the interface device1403 can also be downloaded and stored in firmware on the interfacedevice. The classifier, parameters, configuration, training data, andother settings can be updated on-the-fly.

In some cases, the user interface device 1403 can include additionalmechanical and electronic features to improve the performance of theuser interface system 1400. For example, mechanical features can beprovided to help to assure a predetermined location and orientation ofthe user interface device on the user so that, for example, a set ofelectrodes reliably face the anterior side of the wrist with respect tothe radial compartment of the anterior wrist. In some instances, theelectrodes can provide two channels of signals from the wrist, the firstchannel from the radial compartment and the second channel from thetarget flexor digitorum profundus to enhance the robustness of theinformation available for classification.

In some examples, a second sensor or set of sensors could be included onthe user interface device positioned and oriented to face the posteriorwrist on the radial side to detect and classify tissue electricalsignals associated with extension movements of one or more fingers. Thesignals from the second sensors can be used to enhance the quality ofthe classification of mouse events and to enable classification of otheroccurrences related to the mouse.

In some cases, the user interface device can be implemented asimplantable sensors in the user's wrist or tattoo-based or sticker-basedflex circuits that would conform to the user's skin, or combinations ofthem.

In some implementations, other types of sensors can be included as partof the user interface device, the physical device, or other componentsof the user interface system 1400. For example, capacitive sensors couldbe used to confirm when the user's hand is touching or resting on themouse. In some embodiments, the interpreter would not predict a mouseclick when the user's hand is not touching or resting on the mouse. Insome cases, the sensors could include proximity sensors orphoto-resistors. Such other sensors and components can be held on thebody of the user in the vicinity of the user interface device or off thebody. Signals from the sensors can provide additional information foruse in classification of events or actions intended by a user.

In some implementations, the user interface device can be held on theforearm above the wrist including near the wrist, near the elbow, or inbetween. The farther the user interface device is held above the wrist,the greater the possible time difference between the tissue electricalsignals and the corresponding physical mouse click, potentiallyenhancing the beneficial effects mentioned earlier.

Although the user interface device can, in some implementations, be adistinct “standalone” device, in some cases, some or all of the featuresand functions of the user interface device can be integrated in anotherdevice, such as a bracelet, a watch, or a mouse, or combinations ofthem. In some cases, functions and features of the user interface devicecan be incorporated directly into the physical mouse either in itsoriginal construction or as an add-on, and the two can be coupled bywired or wireless connections.

In some instances, sensors such as proximity sensors can be integratedinto a physical mouse to detect whether contact is being made by a userwith the device. The signals from the integrated sensors can be providedto the interpreter for use in confirming that tissue electrical signalsclassified as mouse clicks, for example, are likely accurate because theuser is actually touching the mouse.

In some implementations, a physical mouse click may not be necessary inorder for the interpreter to predict a mouse click accurately. In otherwords the user may be touching the mouse and may send signals from herbrain associated with an intent to click the mouse, but the signals neednot result in a physical mouse click for the interpreter to correctlyinterpret that a mouse click is intended.

In some cases, the user interface device and a physical mouse could bephysically or electrically (or both) interconnected and could share dataconnections to the interpreter, which can reduce the need for multiplecables and improve transmission speeds.

In order to optimize the accuracy of the prediction of a user physicalmouse click by the techniques described above, it can be desirable tocalibrate and retrain the classifier using the occurrences and timing ofthe physical mouse clicks and the signals from the user interfacedevice. By acquiring a stream of data about tissue electrical signalsand labeling segments of the data as corresponding to physical mouseclicks, training data can be generated that can be used to update thetraining of the classifier. This calibration updating process can beexecuted continuously as the user interface device and the physicalmouse are being used in a live gaming context.

In some cases, calibration and training of the classifier can be done bytraining software 1422 dedicated to that function. The training softwarecould include a game (such as electronic target practice or skeetshooting) that involves frequent physical mouse clicking. A stream ofdata taken from the user interface device while a particular user playsthe game then can be labeled by data representing physical mouse clicks.The labeled data can then be used to train the classifier for use bythat user.

The data streams received by and processed by the interpreter willdepend on the particular type of physical mouse being used, theparticular behavior of the individual physical mouse, the physical andelectrical characteristics of the user interface device, its location onthe user, the style of mouse click and use by particular user, theparticular game being played, and a wide variety of other factors. It isuseful to calibrate and train the classifier to take account of suchfactors. Effective training and retraining of the classifier canoptimize its effectiveness in correctly predicting the existence andtiming of physical mouse clicks.

In some instances, effectiveness measurement software 1424 can beexecuted on data streams derived from operation of the user interfacedevice and a particular physical mouse by the user. Metrics can bedevised to measure the effectiveness of the use of the user interfacedevice in providing accurate predictions of the physical mouse clicks.The metrics can be reported to the user along with high level simpleinterpretations such as “This device may not provide any benefit inaccelerating your game play.” In some examples, users may not benefitbecause of their physiology, the placement of the user interface device,their preference of mouse type, or other factors. As a result of suchfactors, the physical mouse clicks may be delivered sooner than thetissue electrical signals can be classified.

In some implementations, position software 1426 determines the positionand orientation of the user's hand relative to the physical mouse, whichenhances the ability of the classifier to reduce false positivepredictions and false negative predictions of the existence of physicalmouse clicks. The position software can use signals from sensors,cameras, gyroscopes, and accelerometers, and combinations of them, onthe user interface device, the physical device, or the recognitiondevice, or combinations of them, for this purpose.

The level of mouse activity can also be a relevant factor in thedetermination of position and orientation. Among the sensors that couldbe used are force sensitive resistors, IMUS, proximity sensors,capacitive sensors, and photoresistors, and combinations of them. Thehand position and orientation can be determined continuously during useof the user interface device and mouse, and the results used as inputsto the classifier, for example. In some cases, the information can beused to provide feedback to coach the user on more effective handpositions and orientations.

In some cases, the user interface device, its interaction with thephysical mouse, and the interpreter can be configured by or for a givenuser to customize their use and operation for that user and to affecthow they respond to the user during a gaming session. For example, thesensitivity of the classifier to the actions of the user can be set topredict the occurrence of physical mouse clicks at a selected degree of“hardness” of the mouse click, such as a hard click or a soft click,depending on the tissue electrical signals captured. The classifiercould also be configured to recognize different types of motion offingers of the user as representing physical mouse clicks or otherphysical device actions, such as contractions of muscles (or extensions)of the thumb, the middle finger, the index finger, or the pinky finger,or combinations of them. In some applications, these motions aredifferent from or unrelated to physical mouse clicks. For example, theinterpreter can be configured to predict a physical mouse click when theuser extends her index finger without regard to whether the extensionachieves a physical mouse click and without regard to whether a physicalmouse is present in the context.

In some examples, the classifier can not only predict the existence andtiming of physical mouse clicks, but also determine how hard the user ispushing (the pressure) on the physical mouse button. The ability totrack the degree of pressure on the mouse button can provide usefuladditional information about intended actions. Such information can beused to provide a richer, more complex, and more subtle range and typesof interactions between the user and the game than is possible withsimple binary mouse clicks.

In some cases, the inclusion of one or more IMUS in the user interfacedevice, the physical device, or other components of the user interfacesystem can be used to track subtle wrist rotations and other motions onthe physical device. Such subtle motions can be mapped to gesturesrelevant to a context of a game. A user of a game typically is not onlyperforming mouse clicks but also voluntarily or involuntarilymanipulating her arm, wrist, or fingers in ways to indicate or implyactions that are relevant to the game. Such manipulations can beinterpreted by the classifier as, for example, a “lean” gesture or a“strafe” gesture.

IMU data also can be used to silence or otherwise affect a mouse clickor other physical action on a physical device. In some cases, the IMUsignals may be combined with activation signals to enable the classifierto predict a mouse click. In addition, combinations of data from the IMUand other sources can be used by the classifier to determine a varietyof different mouse actions (left click, right click, middle click, thumbbuttons, and mouse cursor motion, among other things). A gaming mouseimplemented in this way could have up to 20 programmable keys.

In some cases, the user interface device or the physical mouse or bothcan provide feedback to a user with respect to mouse clicks. Forexample, the feedback could confirm physical mouse clicks or theduration of physical mouse clicks or both. The feedback can also confirmmouse down and mouse up events separately.

In some examples of the operation of the system, the following sequenceof hardware and software interactions can occur.

Biopotential signals are collected at the anterior portion of the wristand delivered to local circuitry on the user interface device. Thesignals can be processed at the user interface device or forwarded intheir raw state to the interpreter or a combination of the two. Thesignals can be carried over Bluetooth low energy channel to theinterpreter. In some cases, combinations and sequences of biopotentialsignals are classified on board the user interface device and theclassifications (indicating, for example, a mouse click) are sent to theinterpreter. In some instances, combinations and sequences ofbiopotential signals are classified in the interpreter.

The interpreter receives two streams of data with respect to mouseclicks. One stream is derived from the biopotential signals collected atthe user interface device. The other stream is received from thephysical mouse button switch. In each of the two streams of mouse clickdata, each of the mouse click events is time stamped. By analyzing thetimestamps for the respective mouse click events in the two streams ofdata, the interpreter can match corresponding mouse click events in thetwo streams. Often, for example, the mouse click event in the userinterface device during will occur slightly ahead of (for example in arange around 30 ms) the mouse click event in the physical mouse clickstream. Two corresponding mouse click events are paired to form aclick-pair. The click-pair is analyzed by the interpreter to determinewhich mouse click event occurred first. The interpreter then posts theearlier-to-occur mouse click event of the click-pair to the event queueof the operating system where it is treated as a mouse click.

In addition, the time stamped streams of data are stored and used foranalytical purposes. For example, the data can be used to calculate userperformance metrics, can be used for training the classifier, and can beused for development and improvement of the system, among other things.

As discussed earlier, before the user interface device is ready for use,a user calibration can be performed to optimize the performance andreliability of the classifier as applied to a particular user. Varioustechniques can be used for calibration including the following. Thesystem can be continuously calibrated dynamically based on currentlycollected data streams from the user interface device and the physicalmouse. The system could be calibrated using a dedicated calibrationroutine 1428 run prior to the actual use of the user interface device.The dedicated calibration routine could be specifically designed forthat purpose. In some cases, a user can make adjustments and additionsdirectly to suit her intentions.

In general, mouse click event data from the user interface device andfrom the physical mouse as well as calibration data, calibrationperformance, and quality metrics can be provided directly to the user invarious forms. Among other things, this can help the user to improve heruse of the system, including improving average speed benefit relative tostandard physical mouse clicks.

Processing of streams of mouse click event data, developing anddistributing classifier and other machine learning software, managinguser profiles, and a variety of other functions and features can beprovided either from a cloud-based server system or through aclient-based system located with the user, or combinations of the two.

In some cases, multiple classifiers could be maintained and operated onthe interpreter and the user could be given the opportunity to choosewhich classifier she wishes to have applied to her use of the userinterface system. The user could also determine custom preferences, forexample the threshold for the determination of whether a mouse click isa hair-trigger mouse click or a heavy-handed mouse click.

Although much of the discussion above has been focused on the use of thephysical mouse, mouse clicks, and the attachment of the user interfacedevice for sensing tissue electrical signals at the anterior side of thewrist, the techniques that are described can be applied to a widevariety of other devices in contexts. For example, they can be appliedwith respect to any physical device that can be physically manipulatedby a user, for example, by one or more fingers or wrist of the user. Theuser interface device can sense tissue electrical signals at a varietyof other places on the arm. Although gaming has been the focus of someof the examples discussed above, the techniques can be applied in a widevariety of contexts and applications, including industrial andcommercial applications and in the control of a wide variety ofproductivity, design, and management applications.

Referring to FIG. 3, a pair of smart glasses is another example of aresponsive device that can be controlled, commanded, informed, orotherwise operated using a user interface, such as the user interfaces100 of FIGS. 1A and 1B. In order to operate the smart glasses, thesecond interpreter 106B may include specialized interpretations 302A-Dparticularly suited to operation of smart glasses. As before, the secondinterpreter receives electrical signals or data 126 representative ofinputs 114 from the human 110 and interprets the electrical signals ordata 126 to generate outputs 116A-D for the smart glasses (responsivedevice 112). In this example, the outputs 116A-D for the smart glassesare commands to perform one or more actions 122. For example, if thesecond interpreter 106B classifies the electrical signals 126 as an“Intended Index Finger Flick Up” gesture 302A, an “Activate Glass”command 116A is generated, and the smart glasses perform thecorresponding action 122. If the second interpreter 106B classifies theelectrical signals 126 as a “Clockwise Palm Rotation” gesture 302B, a“Move Forward in Timeline” command 116B is generated, and the smartglasses perform the corresponding action 122. If the second interpreter106B classifies the electrical signals 126 as a “Counter-clockwise PalmRotation” gesture 302C, a “Move Backward in Timeline” command 116C isgenerated, and the smart glasses perform the corresponding action 122.If the second interpreter 106B classifies the electrical signals 126 asa “180 Degree Palm Rotation & Hold” gesture 302D, a “Go Back to TimelineScreen” command 116D is generated, and the smart glasses perform thecorresponding action 122. Thus, the user (human 110) can provide input114 to operate the smart glasses (responsive device 112) without movinghis hands toward the smart glasses or touching the smart glassesdirectly. Compared to user interfaces that include a touchpad that isphysically located on the smart glasses, the user interface describedhere has the advantages of improved ergonomics, energy efficiency, andtime savings. Control of the responsive device can be rapid, simple,easy, and effective.

FIG. 4 shows an example on-screen display 400 of a pair of smart glasses(responsive device 112). The on-screen display 400 includes visualfeedback icons 402A-D that indicate to a user that a particular output116A-D for the smart glasses has been generated. For example, referringback to FIG. 3, if an “Activate Glass” command 116A is generated, visualfeedback icon 402A may indicate this to the user (human 110) by changingcolor, flashing on and off, increasing in size, or giving any othervisual indication. If a “Move Forward in Timeline” command 116B isgenerated, visual feedback icon 402B may indicate this to the user. If a“Move Backward in Timeline” command 116C is generated, visual feedbackicon 402C may indicate this to the user. If a “Go Back to TimelineScreen” command 116D is generated, visual feedback icon 402D mayindicate this to the user. While visual feedback icons are describedhere in the context of smart glasses, it is understood that these iconscan be included on the on-screen display of any responsive device 112 toinform a user when a particular output has been generated. In somecases, audio or haptic indicators can be included in addition to, orinstead of, the visual feedback icons described in relation to FIG. 4.For example, each of the commands 116A-D may be associated with a uniqueaudio output that is produced by the responsive device 112 when thecommand is generated. In some examples, each of the commands 116A-D maybe associated with a unique vibration pattern of a haptic feedbackelement included within the body of the responsive device 112.

Control of the wide variety of other features and functions of smartglasses or applications running on smart glasses can be achieved by oneor more user interfaces that rely on nerve or other tissue electricalsignals.

Referring to FIG. 5, a smartwatch 500 is an example of a responsivedevice 112 that can be controlled, commanded, informed, or otherwiseoperated by a user interface such as the user interfaces 100 of FIGS. 1Aand 1B. Notification management and information accessibility are keyfeatures of smartwatch designs. A smartwatch 500 can pair with anotherwireless device, such as a smartphone, and pass information to the userusing tactile, visual, auditory, or other sensory cues, or a combinationof them. For example, as shown in FIG. 5, the on-screen display 506 ofthe smartwatch 500 shows a visual icon 502 indicating that a new messagehas been received. The on-screen display 506 also shows two selectablebuttons 504A and 504B, giving the user the option to open the message ordismiss the notification. Currently, the “Open” button 504A is toggledas indicated by thickened borders.

Applying a user interface such as the user interfaces 100 of FIGS. 1Aand 1B in this context, if a user (human 110) desires to open themessage, he can flick his index finger up. The nerve or other tissueelectrical signals from this motion (input 114) are interpreted by afirst interpreter 106A located in a sensor 124 to generate electricalsignals 126 representative of the detected nerve or other tissueelectrical signals. The generated electrical signals 126 are thenclassified by a second interpreter 106B located in a controller 128 asan “Intended Index Finger Flick Up” gesture 108B, and an output 116 isgenerated to cause the smart watch (responsive device 112) to select thetoggled “Open” button 504A (action 122). Alternatively, if the userdesires to dismiss the notification, he can rotate his palm clockwise totoggle the “Dismiss” button 504B and then perform an index finger flickup to cause the smartwatch 500 to perform a selection action.

In addition to opening and dismissing notifications, a user interfacesuch as the user interfaces 100 of FIGS. 1A and 1B can be extended torespond to notifications as well as navigate and operate the variousmenus and applications run by the smartwatch 500. Thus, the user (human110) can provide input 114 to operate the smartwatch (responsive device112) without using his opposite hand (the one not wearing thesmartwatch) or touching the smartwatch screen 506 directly. Compared touser interfaces that require touch control, the user interface describedhere has the advantages of improved ergonomics, energy efficiency, andtime savings, among other benefits.

Referring to FIGS. 10 and 11, a camera is an example of a responsivedevice 112 that can be controlled, commanded, informed, or otherwiseoperated using a user interface such as the user interfaces 100 of FIGS.1A and 1B. In some cases, the camera can be a standalone device such asa hand-held camera, a remote camera, or a security camera. In somecases, the camera can be included in or connected to a camera-compatibledevice such as a smart phone, a drone, a pair of smart glasses, or anyother camera-compatible responsive device. While the examples thatfollow describe the generation of direct outputs for the camera, in somecases, outputs may be generated for an intermediary device such as acamera-compatible device that runs an application for controllingactions of the camera.

In order to operate the camera, the second interpreter 106B may includea specialized set of interpretations 1002A-D appropriate to the camera.The second interpreter 106B receives electrical signals 126representative of inputs 114 from the human 110 and interprets theelectrical signals 126 to generate outputs 116E-H for the camera(responsive device 112). In this example, the outputs 116E-H for thecamera are commands to perform an action 122 when the camera is in anactive state. The active state of the camera can be presented on anon-screen display, showing, for example, an active camera default screen1100A (shown in FIG. 11). In some cases, the on-screen display can be ascreen on or connected to a standalone camera or a camera-compatibledevice. For example, if the second interpreter 106B classifies theelectrical signals 126 as an “N-Degree Palm Rotation” gesture 1002A, a“Zoom In/Out” command 116E is generated, and the camera performs thecorresponding action 122. In some cases, the magnitude of N cancorrespond to a zooming magnitude, and the direction of the palmrotation can correspond to either a zoom in or zoom out action 122. Forexample, referring to FIG. 11, on-screen display 1100B illustrates azooming function. The zoom screen 1100B has a “+” icon 1106 located onthe right half of the screen, a “−” icon 1108 located on the left halfof the screen, and a numerical zoom multiplier 1110. In this example, ifa user (human 110) rotates her palm clockwise, the camera will perform a“Zoom In” action. If she rotates her palm counterclockwise, the camerawill perform a “Zoom Out” action. If the second interpreter 106Bclassifies the electrical signals 126 as an “Intended Index Finger FlickUp” gesture 1002B, a “Take Photo OR Start/Stop Video” command 116F isgenerated, and the camera performs the corresponding action 122. If thesecond interpreter 106B classifies the electrical signals 126 as an“Index Finger Lift and Hold” gesture 1002C, a “Switch Between Camera andVideo” command 116G is generated, and the camera performs thecorresponding action. Referring to FIG. 11, in response to the “SwitchBetween Camera and Video” command 116G, an on-screen display affected bythe camera may switch between screens 1100A and 1100B. In the activecamera default screen 1100A, the crosshairs 1102 are circular. In theactive video default screen 1100C, the crosshairs 1104 are rectangular,and a recording time indicator 1114 is displayed in the lower right handcorner of the screen 1100C. If the second interpreter 106 B classifiesthe electrical signals 126 as a “Fist Formation” gesture 1002D, a“Display Menu” command 116H is generated, and the camera performs thecorresponding action. Referring to FIG. 11, in response to the “DisplayMenu” command 116H, an on-screen display affected by the camera maydisplay menu screen 1100D containing menu 1112, enabling the user tocontrol camera features such as brightness or flash. While certainpredetermined gestures, predetermined actions, and correspondencesbetween gestures and actions are described, any number or types ofgestures, actions, and correspondences between gestures and actions thatenable or assist the control of responsive devices may be used. Forexample, additional actions that enable or assist the control of acamera or a camera-compatible device may include swapping betweencameras (e.g. swapping between a front and a rear camera), addingfilters, and implementing augmented reality features such as Emojis.

Thus, the user (human 110) can provide input 114 to operate the camera(responsive device 112) without moving her hands toward the camera ortouching the camera directly. Compared to user interfaces that requiretouching the camera, the user interface described here has theadvantages of improved ergonomics, energy efficiency, and time savings,among other benefits. Furthermore, the user interface described inrelation to FIG. 10 is compatible for applications where the camera maynot be in reach of the user, such as cameras mounted on flying drones orremotely operated cameras.

In some cases, a user interface such as the user interfaces 100 of FIGS.1A and 1B can be used to replicate the functionality of user interfacedevices 102 such as real or virtual keyboards. FIG. 6A shows a virtualkeyboard 600 that may be presented on an on-screen display of aresponsive device 112. A vertical selector line 602 moves repeatedlyback and forth between the left and right edges of the virtual keyboard600, and a horizontal selector line 604 moves repeatedly up and downbetween the top and bottom edges of the virtual keyboard 600. To selecta letter, a user can perform two index finger flicks up. The first indexfinger flick up generates nerve or other tissue electrical signals(input 114) that are interpreted to generate an output 116 to theresponsive device 112 commanding it to stop the motion of the verticalselector line 602 (action 122) at a desired position that overlays adesired key. The second index finger flick up generates nerve or othertissue electrical signals (input 114) that are interpreted to generatean output 116 to the responsive device 112 commanding it to stop themotion of the horizontal selector line 604 (action 122) at a desiredposition that also overlays the desired key. Once the vertical selectorline 602 and the horizontal selector line 604 are both stopped, theletter that is closest to the intersection 606 of the two lines isselected, and a key code representative of that letter is sent to theresponsive device 112. The vertical selector line 602, the horizontalselector line 604, or both begin to move again, and the process isrepeated.

Although FIG. 6A illustrates an alphanumeric QWERTY keyboardarrangement, a similar technique can be used for any kind of keyboard orany kind of arrangement of letters or numbers or other symbols to bechosen by user. Such a keyboard or other arrangement could be smaller,larger, of a different pattern or configuration, or include fewer or agreater number of letters, numbers, or symbols.

Referring to FIG. 6B, the functionality of a keyboard (user interfacedevice 102) can be replicated using groupings of letters 608A-D that mayappear on the on-screen display of a responsive device 112. Currently,the second grouping from the left 608B is selected as indicated bythickened borders. To select a letter, a user (human 110) can navigateto the grouping containing the desired letter by rotating his palm. Thismotion (input 114) is interpreted by the user interface 100 to generatean output 116 for the responsive device 112 commanding it to togglethrough the other groupings in a similar fashion to the one describedearlier. To select a grouping, the user can perform an index fingerflick up. Once a grouping is selected, the user can toggle through theletters within the grouping by rotating his palm. To select a letter,the user can perform an index finger flick up. At any time, if the userdesires to view all of the groupings 608A-D, he can rotate his palm 180degrees so that it is facing upward and hold his palm in that positionfor a certain period of time, for example, for 1-3 seconds. Variousgroupings of letters 608A-D may be used. In some cases, letters may begrouped such that the most likely letters to be selected are organizedin a manner that minimizes the expected amount of motion made by theuser.

In some cases, a user interface such as the user interfaces 100 of FIGS.1A and 1B may be used to operate multiple responsive devices 112A-D.Referring to FIG. 7A, a wrist-worn user interface device 102 may becapable of connecting wirelessly (e.g. via Bluetooth or other wirelesscommunications within a frequency range of about 2400 MHz to about 2500MHz) to a computer 112A, a smartphone 112B, a pair of smart glasses112C, a smartwatch 112D, or one or more other devices, or combinationsof them simultaneously or essentially simultaneously. In some cases,communicating to multiple responsive devices 112A-D can be managed usinga Bluetooth round robin technique. In a Bluetooth round robin, the userinterface device 102 interacts with each responsive device in turn for aset amount of time. For example, a Bluetooth connection 700A may firstbe established between the user interface device 102 and the computer112A for one minute, allowing the user to operate the computer 112A.After one minute, the Bluetooth connection 700A is terminated, and a newBluetooth connection 700B is established between the user interfacedevice 102 and the smartphone 112B, allowing the user to operate thesmartphone 112B. After another minute, the Bluetooth connection 700B isterminated, and a new Bluetooth connection 700C is established betweenthe user interface device 102 and the smart glasses 112C, allowing theuser to operate the smart glasses 112C. After another minute, theBluetooth connection 700C is terminated, and a new Bluetooth connection700D is established between the user interface device 102 and thesmartwatch 112D, allowing the user to control the smartwatch 112D.Finally, after another minute, the Bluetooth connection 700D isterminated, a new Bluetooth connection 700A is established between theuser interface device 102 and the computer 112A, and the cycle repeatsitself. Although this example emphasizes Bluetooth connections, asimilar round robin scheme can be implemented with wireless connectionsof any kind.

In some cases, communicating from a user interface to multipleresponsive devices can be managed using an intermediary routingcomputing device 702, as shown in FIG. 7B. In this configuration, a userinterface device 102 maintains a continuous wireless connection (e.g., aBluetooth connection or other wireless connection communicating within afrequency range of about 2400 MHz to about 2500 MHz) to the intermediaryrouting computing device 702, and the intermediary routing computingdevice 702 is responsible for establishing and terminating connections706A-D with the computer 112A, the smartphone 112B, the smart glasses112C, and the smartwatch 112D, for example. In some instances, theintermediary routing computing device 702 can be configured always toprioritize a wireless connection 700A with the computer 112A overwireless connections 700B-D with the other devices 112B-D. In someexamples, the intermediary routing computing device 702 can beconfigured to automatically establish a connection with the responsivedevice that is displaying the most recent notification. In an example,the user may be able to manually select which responsive device 112A-Dhe would like to connect to.

In some cases, the intermediary routing computing device 702 is capableof managing a variety of different wireless connections and datatransfer protocols (e.g. UDP, TCP, etc.) that may be specific to eachresponsive device 112A-D. In such a configuration, the user interfacedevice 102 only needs to establish one connection and one data transferprotocol with the intermediary routing computing device 702. Meanwhile,the intermediary routing computing device 702 handles forwarding thedata or corresponding commands from the user interface device 102according to the specific connection type and data transfer protocol ofeach responsive device 112A-D. For example, a user interface device 102may send electrical signals 126 representative of nerve or other tissueelectrical signals 114 to a smart speaker device (intermediary routingcomputing device 702). From there, the smart speaker device can manageconnections to all other responsive devices 112A-D, either by forwardingthe raw digitized electrical signals 126, higher level interpretationsof the signals, or corresponding command outputs to one or more of theresponsive devices 112A-D. This technique can be expanded such that someuser interface configurations may include multiple intermediary routingcomputing devices 702, with each intermediary routing computing device702 serving as a node in a mesh network. In such a configuration, rawdigitized electrical signals 126, higher level interpretations of thesignals, or corresponding command outputs can be distributed to abroader physical area than would be possible with a single directBluetooth, Wi-Fi, or other wireless connection.

In some applications, it may be possible to use a single user interfacedevice to control two or more responsive devices simultaneously andwithout conflict by defining, for example, mutually exclusive easilydistinguishable gestures to be applied specifically and only tocorresponding respective responsive devices. For example, finger flickscould be used to control a computer display while palm rotations couldbe used simultaneously to control a robotic hand. In some applications,the two or more responsive devices being controlled simultaneously canhave a shared user interface operated by the single user interfacedevice. For example, the user interface device may be configured toselect and drag media or files from one device to another, such asbetween two laptops, or between different devices like phones andtelevisions.

In some cases, a user interface such as the user interfaces 100 of FIGS.1A and 1B may be distributed across two or more user interface devices102. For example, referring to FIG. 8, a user (human 110) may be wearingtwo wrist-worn user interface devices 102A, 102B, each of which is ableto generate electrical signals or data 126A, 126B representative ofinputs 114 from the respective hand of a human 110. In some cases, themultiple user interface devices 102 may be distributed across multiplebody parts or across multiple users. When the generated electricalsignals or data 126A, 126B arrive at the second interpreter 106B of thecontroller 128, there are two possibilities for generating outputs 116for the responsive device 112. In some cases, the electrical signals ordata 126A, 126B are interpreted simultaneously as two discrete inputs,and two outputs 116A-B are generated for the responsive device 112corresponding to the two discrete inputs. In some cases, the electricalsignals or data 126A-126B are combined and interpreted as a joint input,generating a single output 116C for the responsive device. A variety ofother arrangements would be possible to provide a robust, rich, deep,quick, easy facility for controlling a responsive device.

For example, if the user is wearing two wrist-worn user interfacedevices 102A-B and is using the user interface devices 102A-B tomanipulate a music queue, she can control the volume of an audio track(output 116A) with one hand while selecting the next song (output 116B)with her other hand. This ability to multitask increases the efficiencyof operating a responsive device 112 by allowing for more outputs to begenerated for the responsive device 112 in the same amount of time.

In some cases, if the electrical signals or data 126A, 126B from the twowrist-worn user interface devices are combined to generate a singleoutput 116C, a wider sample space of interpretations can be identified,allowing for a larger number of output options 116C to be generated forthe responsive device.

Although the examples above referred to these two wrist-worn userinterface devices, the two or more user interface devices could includea combination of any kind of user interface devices providingopportunities for even more subtle, complex, robust opportunities forcontrolling one or more responsive devices.

In some cases, the electrical signals or data 126A, 126B from one ormore user interface devices 102A, 102B may be combined with electricalsignals or data from sources other than user interface devices such asaudio input from a human. Similar to the description given for FIG. 8,in some cases, the electrical signals or data 126A, 126B from the one ormore user interface devices and the electrical signals or data from theother sources are interpreted simultaneously as two discrete inputs,generating two discrete outputs. In some cases, the electrical signalsor data 126A, 126B from the one or more user interface devices and theelectrical signals or data from the other sources are combined andinterpreted as a joint input, generating a single output. For example,if the user is wearing a wrist-worn user interface device (e.g. 102A),she can simultaneously rotate her palm to control the zoom of a camera(generating electrical signals or data 126A from the wrist-worn userinterface device 102A) and say the words “Take Photo” (audio input fromthe human) to cause the camera to take a photograph.

Although the examples above emphasize user interfaces 100 that includecontrollers 128 that perform high level interpretations to classifygestures (e.g. gestures 108A-C), it is understood, as described above,that in some cases, the controller 128 includes an interpreter 106B thatperforms lower level interpretations such as “Digitized Raw Signal”108D, producing an output 116 substantially similar to receivedelectrical signals 126. In the smart speaker example described inrelation to FIG. 7B, the smart speaker (intermediary routing computingdevice 702), in some cases, simply forwards along the digitized rawsignal received by user interface device 702 to responsive devices112A-D. This configuration has the advantage of allowing each responsivedevice 112A-D to perform its own interpretations and actions based onthe digitized raw (e.g. nerve or other tissue electrical) signal. Inthis example, the intermediary routing computing device is consideredboth a user interface device and a responsive device. It is a userinterface device because it is an intermediary structure between a humanand another responsive device, and it is also a responsive devicebecause it performs an action (forwarding the digitized raw signal) uponreceiving a corresponding input signal.

In some implementations, one or more sensors 124 within a given userinterface device 102 are subjected to a calibration process at one ormore times. For example, IMU sensors that track spatial and angularposition and acceleration of an object may become inaccurate due toperforming integration operations based on data collected at discretemoments in time. To reduce this inaccuracy, a reference point can beestablished that is independent of the position of the IMU. Acalibration system that communicates wirelessly with the IMU (e.g.,using wireless protocols or Bluetooth), can transmit values for thereference point, allowing the virtual vectors provided by the IMU to becomputed against the real-world vectors provided by the calibrationsystem. The IMU can be configured to recalibrate its position after aset number of spatial calculations or as commanded by the user or both.

In some cases, the user interfaces are configured to account forlatency, for example varying delivery latency associated with wirelesspacket transmissions. To account for this variation, the electricalsignals 126 representative of inputs 114 from the human 110, such ashand movements, can be signal processed on a time-variable basis. Forexample, if a first transmission packet (e.g. electrical signal 126) isdelivered with a 10 ms latency and a second transmission packet isdelivered with a 5 ms latency, the signal processing of the secondtransmission packet can be performed with a 5 ms delay to cause thetemporal spacing of the output signals to be true to the originaltemporal spacing of the inputs 144 from the human 110.

In some examples, the user interfaces participate in applying securityfeatures to control the use of responsive devices in accordance withauthorizations. For instance, such security features can be used toensure that unauthorized users cannot operate responsive devices withoutpermission. We sometimes use the term “authentication” and“verification” interchangeably to refer, for example, to determining theunique identity of a human or to the process of confirming whether theuniquely identified human has authorization to make full or partialcontrol or use or a user interface device or a responsive device orboth.

In some cases, data derived from nerve or other tissue electricalsignals captured by a user interface device at the wrist in the courseof normal activity (i.e., in a passive mode) or in the course ofoperating a responsive device (i.e., in a usage mode) can be used by anauthentication process. In some instances, other kinds of data gatheredby a wrist-worn device, such as data indicative of position and motionof the wrist, fingers, or arm, can be used in an authentication process.In some cases, nerve or other tissue electrical signals captured by theuser interface device at the wrist can be combined with signals capturedby one or more user interface devices at other parts of the body in anauthentication process. The authentication process can be executed atspecific times, such as when a human begins to use a user interfacedevice or a responsive device, or at repeated times. In some instances,the verification process can be executed continuously in the context ofongoing operation of a user interface device or a responsive device.Authentication may be achieved by communication with a remote centralserver, by neighbor-to-neighbor communication in a mesh network, orother verification approaches, or combinations of them.

Referring to FIG. 9, in some implementations, a user authenticationsystem 900 includes an identification apparatus (device) 902 forinterpreting signals (signals or data or both 904) detected or receivedby the identification apparatus 902 as one or more markers (marker data906) that uniquely denote or characterize an identity 908 of aparticular human 916. The signals are data that can be received from oneor more user interface devices or one or more responsive devices orother sources or combinations of them. For example, the signals can bereceived from one user interface device located on the user's wrist anda second user interface device located on the user's leg. The IDapparatus can use the markers to uniquely distinguish humans based onindividual or sequential amplitude, pattern, timing, or length orcombinations of them, of the detected or received signal or data. Insome examples, the markers can be a binary system such as Morse code orcan be a multidimensional system leveraging different signals or data,for example, from different sources on the human or can be analogamplitudes, timing, or length, or combinations of them, for example. Theidentification apparatus 902 can be connected electronically to one ormore responsive devices or user interface devices or combinations ofthem 910A-B, whether physically or by software, communicating eitherdirectly 912 (paired mode) or through a remote server 914 (cloud-basedmode) or both to control the use of the device based on results of theauthentication process. The identification apparatus 902 may include anactivation (authentication control) function 920 which, upon making averification determination on the identity 908 of the human 916, canperform a variety of control functions to control access by the humanthrough an access gate 918 to user interface devices or responsivedevices 910A-B or to specific functions or features of the devices. Insome cases, the access can be prevented if the verificationdetermination was unsuccessful. In some instances, the access can beunlimited if the verification determination was successful. In someexamples, the access can be limited with respect to time, location,context, or particular features or functions of the devices.

In some cases, a single verification of the identity 908 of the user 916by the identification apparatus 902 can allow for control of multipleresponsive devices (e.g., responsive devices 910A-B) without the needfor any repeated verification. For example, upon the identificationapparatus 902 verifying a user's identity 908 based on biometric signals(e.g., nerve or other tissue electrical signals) collected by awrist-worn user interface device (e.g. user interface device 102), theuser may then be allowed access to any number of responsive devices thathe is authorized to use such as laptops, smart televisions, and mobilephones, for example, without the need to sign into (be authenticatedfor) each responsive device.

In some implementations, the signal or data 904 incorporatesskin-surface-derived nerve or other tissue electrical activity signalsor data, either in isolation or in combination with wrist accelerationor orientation, or other wrist position or motion information, orcombinations of them, as measured by another sensor such as an inertialmeasurement unit (IMU). The signal data 904 and the marker data 906 maycorrespond to repeated, periodic, or discrete time intervals (orinterval-based transformation) or may be derived from continuous signalor marker data, e.g., a continuous function operating in real-time or atmultiple timescales. The signal data may be thresholded, normalized, orcollected as raw data.

As the signal data 904 in a baseline or passive mode (with the userengaging in no activity) can be based on a combination of human finger,hand, and wrist morphology, body composition, and other factors, and asthe signal data 904 generated during an active mode (with the userengaging in an activity that may include motion) involving either aresponsive device 910A-B or the physical world includes additionalelements of, for example, position, velocity, acceleration,three-dimensional orientation, interaction habits, and other featuresand information that are unique to a particular human, the signal data904 (in raw form or transformed into marker data 906), represents areliable and individualized marker (e.g., biomarker). Such a marker hasa variety of practical applications, for example enabling uniqueauthentication of a human as part of a login process when the humanbegins to make use of a user interface device or a responsive device aswell as during ongoing interaction with or use of responsive devices910A-B.

The signal data 904 may be collected in a passive mode (for example,when the user is wearing a user interface device but is not using andhas no intention of operating a responsive device 910A-B). In somecases, signal data 904 may be collected during an active mode duringactivation of the responsive device 910A-B for the purposes of operatingit (after the presentation of a specific prompt to the user to perform asequence of gestures or temporally-arranged activations, whether in anisolated format or an interactive challenge/response format; or withinan overlaid user interface interaction sequence).

The marker data 906 may be unique in isolation, with a one-to-onemapping of each marker data element to the identity 908 of each user916, or may be analyzed for verification purposes in combination withother markers or in combination with other data (such as location, text,symbol, voice, fingerprints, or other information, or combinations ofthem) to create a unique combination marker (multifactor marker).

Any marker may be used by an authentication process for the verificationof a wearer (user) identity 908 when implemented in either a primary(single) or multi-factor manner; for either discrete verification (as amethod for implementing logins, for example) or to enable continuous(adaptive) access during use (as a method for implementation ofreal-time or retroactive access).

The identification apparatus may incorporate various software orhardware-based capabilities to perform on-board or remote-server-basedprocessing of signal data (derived from the measurement of signalsrelating to, for example, the contraction or extension movement of onefinger or multiple fingers, or the movement of the wrist in any axis, orthe movement of the forearm along any axis; or any combination of them)into marker data as described below.

In some implementations, the identification apparatus 902 includes: adetection unit 922 which receives one or more nerve or other tissueelectrical signals (signal data 904) from a surface-based IMU, from oneor more electrodes, or from a combination of those sources; a processingunit 924 which calculates an index value 926 using received signals; anda verification unit 928 which stores index values 926 corresponding topre-defined users 916 and calculates a marker value 906 from athresholded algorithmic correspondence between index values 926delivered from the processing unit 924 and the pre-stored index valuesof users. The processing unit 924 may expose the correspondence dataelectronically to external devices. The processing unit 924 and theverification unit 928 may be housed either in separate devices ortogether within the signal detection unit.

In some implementations, the verification unit 928 incorporates anon-board database apparatus 930 (or, in some cases, an apparatus linkedelectronically to a cloud-based server) which collects biometric andother marker data and which constitutes a resource serving (ortransforming) user-level and aggregated signal data and marker data tothe verification unit 928, or to other verification units.

In some instances, a curation unit 932 enables a single or multipleindependent operators to activate, adjust, and monitor specifiedfunctions of the verification unit; by which the operator mayaccumulate, monitor, and remove selected index values (and otherassociated information) relating to pre-defined users; and by which setsor subsets of the users may be accorded differential access privilegesby the operator, in real-time. Either one or multiple identificationunits 902 may be controlled in such a manner.

In some cases, the curation unit 932 which is housed on-board anidentification apparatus 902 may transfer (on a push basis) selectedsignal data 904 or marker data 906 directly to other identificationapparatuses 934, and store such data as transferred from otheridentification apparatuses 934, in order to facilitate amesh-network-based or otherwise distributed collective verificationscheme. In some instances, there can be a single central curation unitor server.

In some instances, the curation unit 932 incorporates a machine learningand classification system which allows for the development of methods ofassigning markers 906 to signal data 904, as well as development ofmultifactor markers from existing marker data 906 and signal data 904.

User verification may be gated in combination with a method based ondata including the location or proximity of the identification apparatus902 to other verified users.

The signal data 904 and the marker data 906 may be exported from thecuration unit 932 or a remote server, on an individual, aggregated, oranonymized basis for applications separate from verification purposes.

The signal data 904 and the marker data 906 may be used as activationcues for devices connected to the curation unit 932, a remote server, oranother connected system.

When used in continuous mode, a change in the pattern of signal data 904or marker data 906, as identified by the verification unit 928, mayrepresent a trigger for the removal of access for a verified user, as achange in these data may be an indication of a change in the user 916being served by the identification apparatus 902. The system 900 can beimplemented as an exclusive authentication mechanism, or in conjunctionwith other authentication methods as refereed by the user interface orresponsive device being server, for example. In some cases, in additionto authentication (e.g., biometric authentication) using signal data 904and marker data 906, a user may be required to enter a passcode in orderto have access to responsive devices 910A-B. Referring to FIG. 12, auser interface device can be used to enter the passcode for userauthentication. In some cases, a responsive device (e.g. responsivedevices 910A-B) may cause a locked password authentication screen 1200Ato be presented on an on-screen display. A central lock icon 1202Adepicts a closed padlock indicating that the user does not have accessto operate the responsive device. In order to gain access, the user mustenter a four-digit passcode. Similar to the example given with respectto selectable icons 202A-D in FIG. 2, to toggle between the differentnumerical digits, the user may rotate her palm clockwise orcounterclockwise until the digit she intends to select is toggled forselection. To select a toggled digit, the user may perform an indexfinger flick up. If the user enters the correct four-digit passcode, anunlocked password authentication screen 1200B is presented on theon-screen display. On this screen 1200B, a central lock icon 1202Bdepicts an open padlock indicating that the user has access to operatethe responsive device. If the user enters an incorrect four-digitpasscode, the screen remains on locked password authentication screen1200A and the user is prompted to try again. While the example givendescribes a four-digit numerical passcode, passcodes can be of anylength and can comprise alphanumerical characters, shapes, pictures, oricons of any kind. In addition, while a palm rotation gesture and indexfinger flick up gesture are described as examples for entering apasscode, a wide variety of gestures and combinations of gestures andon-screen displays may be implemented to provide a passcode for userauthentication.

As illustrated by the discussion and examples above, in someimplementations, the inputs by a human user to a user interface thatwill be interpreted and then used to control, command, inform, orotherwise operate a responsive device can be one or more predeterminedor predefined inputs such as motions, gestures or intentions. Aninterpreter is expecting to receive such an input, interpret it, andpass along to the responsive device a corresponding data, command,instruction, or guidance that will cause the responsive device toimplement an action. In other words, the inputs and types of input thatthe interpreter can properly interpret and the corresponding actions andtypes of actions to be taken by the responsive device are prearranged.For example, when the user interface device is a wrist worn device thatdetects nerve or other tissue electrical signals corresponding tointended muscle contractions and extensions, and the responsive deviceis a computer display, certain gestures are predetermined as ones thatthe user should use and those gestures correspond to certain actionsthat the user interface devices predetermined to perform in response tocorresponding predetermined gestures.

In some implementations, the predetermined gestures are chosen based onthe capabilities, limitations, and other characteristics of the userinterface device, the capabilities, limitations and othercharacteristics of inputs that the human is capable of or would findeasy, natural, or intuitive to produce, the context in which the userinterface device is used, the ability of an interpreter to interpret thegestures, and the capabilities, limitations and other characteristics ofthe user interface device that is to act based on the gestures, andother factors and combinations of them. The predetermined actions of theresponsive device are chosen based on the capabilities, limitations, andother characteristics of the responsive device, the context in which theresponsive device will act, and the relationship of the actions of theresponsive device to corresponding capabilities, limitations, and othercharacteristics of the user interface device.

In particular applications of the technology that we have described,particular combinations of predetermined gestures of users through theuser interface and predetermined corresponding actions of the responsivedevice are particularly effective, useful, or relevant to the context ofthe application. For example, the gesture of rotating the palm of thehand to cause a displayed object to rotate can be especially intuitive,useful, and effective.

Other implementations are within the scope of the following claims.

The invention claimed is:
 1. A method comprising: receiving firstsignals from a first device representing a state of the first deviceresponsive to a manipulation of a feature of the first device by one ormore fingers of a hand of a user, receiving second signals from a seconddevice distinct from the first device and worn by the user, the secondsignals representing tissue electrical activity associated with nervesor muscles used to perform the manipulation, processing the secondsignals to predict occurrence of the manipulation of the feature of thefirst device by the one or more fingers prior to occurrence of themanipulation, and sending a control signal to an application with whichthe user is interacting, the control signal indicating that manipulationof the feature of the first device occurred, based on the predictedoccurrence of the manipulation from processing the second signals fromthe second device.
 2. The method of claim 1 in which the manipulation ofthe feature and the tissue electrical activity indicative of themanipulation occur at different times.
 3. The method of claim 2 in whichthe processing of the first signals and the second signals comprisesdetermining which of the manipulation of the feature and the tissueelectrical activity more accurately represents the occurrence of themanipulation.
 4. The method of claim 2 in which the processing of thefirst signals and the second signals comprises determining that theearlier of the manipulation of the feature and the tissue electricalactivity more accurately represents the occurrence of the manipulation.5. The method of claim 1 in which sending the control signal to a gameor other application comprises posting an event to an event queue of anoperating system associated with the game or the other application. 6.The method of claim 1 in which the processing of the first signals andthe second signals comprises applying the first signals and the secondsignals to a classifier.
 7. The method of claim 6 comprising trainingthe classifier based on first signals representing manipulations of thefeature and second signals representing tissue electrical activities. 8.The method of claim 7 in which training the classifier comprisestraining the classifier repeatedly while the user is interacting withthe game or other application.
 9. The method of claim 7 in whichtraining the classifier comprises applying a dedicated calibrationroutine before the user begins to interact with the game or otherapplication.
 10. The method of claim 7 comprising receiving instructionsfrom the user about the training of the classifier.
 11. The method ofclaim 1 in which the processing of the first signals and the secondsignals comprises applying the first signals and the second signals to aselected classifier among a set of classifiers.
 12. The method of claim11 in which the selected classifier is selected based on input of theuser.
 13. The method of claim 1 in which the tissue electrical activitycorresponds to a contraction or extension or both of a muscle of theuser.
 14. The method of claim 1 in which the device comprises a mouseand the feature comprises a button or switch of the mouse.
 15. Themethod of claim 14 in which the manipulation comprises a mouse click.16. The method of claim 1 in which the tissue electrical activity occursat the anterior side of the wrist of the user.
 17. The method of claim 1comprising receiving a signal indicative of a state of contact betweenthe user and the first device, and the processing of the first signalsand the second signals to identify the occurrence of the manipulationincludes taking account of the signal indicative of the state ofcontact.
 18. The method of claim 1 in which the first signals arereceived as a stream of samples.
 19. The method of claim 1 in which thesecond signals are received as a stream of samples.
 20. The method ofclaim 1 comprising time stamping the first signals or the second signalsor both.
 21. The method of claim 1 in which two or more channels of thesecond signals are received.
 22. The method of claim 1 comprisingreceiving third signals from sensors facing the posterior wrist on theradial side of the user.
 23. The method of claim 1 wherein the secondcomprises one or more sensors belonging to a tattoo-based orsticker-based component.
 24. The method of claim 1 wherein the seconddevice comprises implanted sensors.
 25. The method of claim 1 whereinthe second device comprises sensors on a watch.
 26. The method of claim1 in which the first device and the second device are electricallycoupled.
 27. The method of claim 1 in which the first device and thesecond device are mechanically coupled.
 28. The method of claim 6 inwhich the classifier is customized for a context in which the game orother application is used.
 29. The method of claim 28 in which thecontext comprises the model or particular unit of the model of the firstdevice.
 30. The method of claim 28 in which the context comprises anidentity of the user.
 31. The method of claim 28 in which the contextcomprises the behavior of the second device.
 32. The method of claim 28in which the context comprises a style of manipulation by the user. 33.The method of claim 1 comprising measuring the effectiveness of the useof the second device by the user.
 34. The method of claim 1 comprisingreporting to the user information about the first signals, the secondsignals, the first device, the processing of the first signals and thesecond signals, or the sending of the control signal to the game orother application, or combinations of two or more of those.
 35. Themethod of claim 1 comprising receiving signals representing a positionor orientation of the hand of the user relative to the first device. 36.The method of claim 1 comprising determining characteristics of themanipulation based on the first signals or the second signals or both.37. The method of claim 36 in which the characteristics comprise theforcefulness of the manipulation.
 38. The method of claim 36 in whichthe characteristics comprise identities of one or more fingers involvedin the manipulation.
 39. The method of claim 36 in which thecharacteristics comprise wrist rotations.
 40. The method of claim 1comprising receiving third signals from an inertial measurement unit andin which the processing takes account of the third signals from theinertial measurement unit.
 41. An apparatus, comprising: a first devicehaving a first output to provide first signals representing a state ofthe first device responsive to a manipulation of a feature of the firstdevice by one or more fingers of a hand of a user; a second devicedistinct from the first device, to be worn by the user, and having asecond output providing second signals representing tissue electricalactivity of the user associated with nerves or muscles used to performthe manipulation; and a responsive device connected to the first outputand the second output to receive, respectively, the first signals andsecond signals, and process the second signals to predict occurrence ofthe manipulation of the feature of the first device by the oner or morefingers prior to occurrence of the manipulation, and send a controlsignal to an application with which the user is interacting, the controlsignal indicating that manipulation of the feature of the first deviceoccurred, based on the predicted occurrence of the manipulation.